The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index; 4 of the 29 sources its claims cite were checked for support.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Open accessAbstract
Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper examines whether strong LLMs can serve as judges for chatbot responses by comparing their judgments with human preferences on two new benchmarks, MT-bench and Chatbot Arena. It finds that GPT-4 can reach over 80% agreement with human preferences, comparable to human-human agreement, while also documenting several judge biases and mitigation strategies.
Key findings
- GPT-4’s judgments aligned closely with human experts on MT-bench: 85% agreement without ties, versus 81% agreement among humans; on Chatbot Arena, GPT-4 also showed high agreement with crowd votes, above 80%.
- Position bias was substantial: with the default prompt, consistency was 65.0% for GPT-4, 46.2% for GPT-3.5, and 23.8% for Claude-v1; swapping positions and few-shot examples raised GPT-4 consistency to 77.5%.
- Verbosity bias appeared in a “repetitive list” attack: Claude-v1 and GPT-3.5 failed 91.3% of cases, while GPT-4 failed 8.7%.
- Reference-guided grading reduced math-judging failures from 14/20 under the default prompt to 3/20, with chain-of-thought at 6/20; self-enhancement was suggested by GPT-4 favoring itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate, but the study could not confirm the bias.
Methods
- The authors created MT-bench, 80 multi-turn questions across 8 categories with 10 questions each, and Chatbot Arena, a crowdsourced anonymous battle platform; they collected about 3K expert votes from 58 labelers on MT-bench and about 30K Arena votes, sampling 3K for analysis.
- They compared LLM judges—mainly GPT-4, GPT-3.5, and Claude-v1—using pairwise comparison, single-answer grading, and reference-guided grading against human expert and crowd votes, measuring agreement and win rates.
- They also evaluated LLaMA and Vicuna variants on MMLU, TruthfulQA, and MT-bench with GPT-4 single-answer grading to test how preference-based and capability-based benchmarks complement each other.
Limitations
- The paper states that it emphasizes helpfulness and largely neglects safety, honesty, and harmlessness; it also notes that helpfulness dimensions such as accuracy, relevance, and creativity are combined into one metric, and its bias mitigations are preliminary.
- The paper reports that self-enhancement bias could not be determined because of limited data and small differences, and that its position-bias test is difficult because the compared answers are very similar and sometimes indistinguishable even to humans.
- The text makes evident that few-shot judging may introduce new biases and makes API calls about 4× more expensive, while zero-shot Vicuna-13B as a judge had low consistency (11.2–16.2%) and high error rates (22.5–78.8%), limiting its use as a scalable open-source judge.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
56 Own result · 35 Own interpretation · 7 Sources not found · 7 No basis given · 3 Partly supported · 2 Own method · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Once aligned with humans, these chat models are strongly preferred by human users over the original, unaligned models on which they are built.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites [31] source
- Partly supportedcites [2] sourceExperimental — accuracy not yet measured
Section 4.5 (p. 21) says crowdworkers clearly preferred the online RLHF models in comparisons with context-distilled and earlier RLHF models. But those preference comparisons do not establish that users strongly preferred them over the original, unaligned base models; the paper discusses base-model improvements on other evaluations (Introduction, p. 4). (LLM, conf=0.86)
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [48]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [14]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that some of the biases are minor or can be mitigated.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that in math/coding category, GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 is still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.3 Win rates under different judges
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, a small high-quality conversation dataset can quickly teach the model a style preferred by GPT-4 (or approximately human) but cannot improve MMLU significantly, as shown by the Vicuna-7B (selected) which is trained with only 4.8M tokens or 3K conversations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most LLM judges favor the first position.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only GPT-4 successfully detected this attack.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As in Table 2, we found all of them exhibit strong position bias.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, it has a high error rate (from 22.5% to 78.8%) because of its weaker instruction-following capability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude-v1 favors itself with a 25% higher win rate.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 12 (Appendix), the few-shot judge can significantly increase the consistency of GPT-4 from 65.0% to 77.5%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plaindirectn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 22
- It may state two separate claims as one. 29
- The quote only partly states this claim. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As LLMs continue to improve, they show potential in replacing human annotators in many tasks [17,20].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3 LLM as a Judge
What the sources say
- Supportedcites [17] source
The Discussion (p. 4) says LLMs could transform text annotation across a variety of tasks and that ChatGPT may already outperform crowd annotators. The Introduction (p. 2) also describes the possibility of annotating entire samples with ChatGPT. These passages support the hedged claim about potential replacement in annotation tasks, particularly those performed by crowd-workers. (LLM, conf=0.88)
- Partly supportedcites [20] sourceExperimental — accuracy not yet measured
The Discussion and Conclusion (Section 5; page unavailable) says ChatGPT showed great potential as a data annotation tool for implicit hateful tweets. It does not establish that LLMs could replace human annotators across many tasks; it also urges caution about incorrect decisions. (LLM, conf=0.86)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
What the sources say
- Partly supportedcites [21] sourceExperimental — accuracy not yet measured
The Introduction (p. 1) presents Dynabench as dynamic data collection with humans and models in the loop, in contrast to static benchmarking. Caveats and Objections (p. 8) explicitly says it anticipates dataset saturation. The copy does not identify overfitting as a challenge of static benchmarks; its discussion of annotators ‘overfitting’ on models (p. 7) describes a potential risk of the dynamic approach instead. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the base LLaMA models showing competitive performance on conventional benchmarks (Table 8), its answers to open-ended questions are often not preferred by humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
- Also stated in: § GPT-4 Judgment: · plain
“Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement ( §4.2, Table 4).”
- Also stated in: § 4.2 High agreement between GPT-4 and humans · plain
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § GPT-4 Judgment: · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The consistency improves significantly from 16.2% to 65.0%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 56.8% when including all three labels, and 85.5% when excluding tie predictions and labels, significantly outperforming random guesses of 33% and 50% respectively, and show positive signals to match GPT-4 (66% and 87% respectively).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite GPT-4 knows how to solve the question (if asked separately), it made a wrong judgement saying both assistants' wrong answers are correct.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 14:
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the classification formulation, every output is recognizable (error rate 0%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 15, Vicuna-13B has a consistency rate from 11.2% to 16.2% across different prompt templates, much lower than all the closed-sourced models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Figure 13 (Appendix), we present an example of an elementary math question in which GPT-4 makes an incorrect judgment.It's worth noting that although GPT-4 can solve the problem (when asked separately), it was misled by the provided answers, ultimately resulting in incorrect judgment.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 3 shows the failure rate of LLM judges under this attack, demonstrating that all LLMs may be prone to verbosity bias though GPT-4 defends significantly better than others.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 11, the position bias is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLMs can achieve an agreement rate of over 80%, on par with the level of agreement among human experts, establishing a foundation for an LLM-based evaluation framework.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 7 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing GPT-4 and other LLM judges, we find they reach a similar non-tie agreement ratio between humans but the number of non-tied votes from GPT-4 is much larger.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that fine-tuning on high-quality dialog datasets (i.e., ShareGPT) can consistently improve the model performance on MMLU and the improvement scales with fine-tuning data size.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-3.5 and Claude-v1 show a verbosity bias towards the longer and repetitive answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both tables, GPT-4 with single-answer grading matches both pairwise GPT-4 and human preferences very well.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On math and coding questions, LLM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the longer prompts make API calls 4× more expensive.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, even with the CoT prompt, we find that in many cases LLM makes exactly the same mistake as the given answers in its problem-solving process (See example in Figure 15 (Appendix), suggesting that LLM judge may still be misled by the context.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar pattern from other LLM judges such as Claude/GPT-3.5.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the pairwise comparison may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically;”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks -benchmarks like MMLU [19] and HELM [24] cannot effectively tell the difference between these aligned models and the base models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [24]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This misalignment of conventional benchmarks underscores the core problem driving this paper: the need for a robust and scalable automated method to evaluate LLM alignment with human preferences.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“single answer grading may be unable to discern subtle differences between specific pairs, and its results may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Conversational benchmarks, like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.However, the diversity and complexity of their questions often fall short in challenging the capabilities of the latest chatbots.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 2.1 Motivation
What the sources say
- Not matchedcites [35]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, high consistency may not imply high accuracy and we are not sure whether the few-shot examples will introduce new biases.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that GPT-4 is more affirmative and less suffered from position bias but other models also perform well when they give an affirmative answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“LLMs are known to have limited math and reasoning capability [10], which results in its failure of grading such questions because they do not know the correct answers.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They focus on different aspects of a model and the recommended way is to comprehensively evaluate models with both kinds of benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To invite participants, we obtained their consent by letting them sign an application form.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.1 MT-bench human evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another more aggressive approach is to assign positions randomly, which can be effective at a large scale with the correct expectations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 8, no single benchmark can determine model quality, meaning that a comprehensive evaluation is needed.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This bias is not unique to our context and has been seen in human decision-making [3,34] and other ML domains [22,41].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matchedcites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [41]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Chain-of-thought is a widely used technique to improve LLM's reasoning capability [47].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
What the sources say
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our LLM-as-a-judge approach can automate and scale platforms of this nature.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that using LLM-as-a-judge to approximate human preferences is highly feasible and could become a new standard in future benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Users are required to accept the terms of use, which obtain their consent and give us the right to release the conversation data.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.2 Chatbot Arena
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These responses are rendered as natural languages and are difficult to be parsed automatically, making the model less useful in a scalable and automatic evaluation pipeline.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-as-a-judge offers two key benefits: scalability and explainability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.2 Advantages of LLM-as-a-Judge
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that this test is challenging because the answers are very similar and occasionally indistinguishable even to humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The single-answer grading can be converted into pairwise comparison results for calculating the agreement.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Honesty and harmlessness are crucial for a chat assistant as well [2].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 6 Discussion
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The paper relies on this without support.
LLMs trained with RLHF already exhibit strong human alignment.
“Because these models are often trained with RLHF, they already exhibit strong human alignment.”
The paper relies on this without support.
There has not been a systematic study of the LLM-as-a-judge approach.
“However, there has not been a systematic study of this approach.”
The paper relies on this without support.
Human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.
“While largely overlooked by existing LLM benchmarks, human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.”
The paper relies on this without support.
The human labelers are mostly graduate students and are considered experts and more skilled than average crowd workers.
“The labelers are mostly graduate students so they are considered experts and more skilled than average crowd workers.”
The paper relies on this without support.
Human evaluation is the gold standard for assessing human preferences.
“While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly.”
The paper relies on this without support.
Model services such as GPT-4 can become expensive with a growing number of evaluations.
“However, model services such as GPT-4 can also become expensive with a growing number of evaluations.”
The paper relies on this without support.
Popular open-sourced LLMs, e.g. Vicuna-13B, show strong language understanding capability and are much cheaper than close-sourced LLMs.
“On the other hand, popular open-sourced LLMs, e.g.Vicuna-13B shows strong language understanding capability, and are much cheaper than close-sourced LLMs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“To mitigate stochastic variance and”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Zheng et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Others employ the LLM-as-a-judge methodology [36], yet these efforts lack both a comprehensive framework design and verification of human consistency.”
From DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) · cites this paper as [36]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“LLM-based evaluators are mainly used in well-defined tasks that require simple information extraction and classification, which has been shown to have high precision Zheng et al. (2023).”
From TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (2024) · cites this paper as Zheng et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The choice of judge can shift accuracy by 5-10 percentage points on identical retrieval output (Zheng et al., 2023).”
From SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additional biases further complicate assessment due to inherent limitations in LLMs: length bias (favoring unnecessarily long outputs), position bias (order effects in candidates), and style bias (favoring certain linguistic styles over content) [337-338,343], potentially distorting results regardless of model capability.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While this method improves scalability and efficiency, it may also inherit biases present in the training data, such as position and verbosity bias.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While automated and scalable, evaluation results can be affected by several factors [337-338], including prompt sensitivity, demonstration setup, and data contamination.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“modern LLMs exhibit diverse generation behaviors in terms of verbosity, reasoning style, and response length.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although the 3-vote majority consensus protocol reduces stochastic fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family (Zheng et al., 2023;Ye et al., 2024), and may be more lenient on verbose near-misses than a human annotator would be (Zheng et al., 2023;Ye et al., 2024;Chen et al., 2024).”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The LLM-as-a-judge framework using GPT-5-mini as evaluator (Zheng et al., 2023) introduces its own sources of variance.”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges [16] without validation on their credibility in reviewing scientific research.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, concurrent works often omit a rigorous backward pass or 'verification' and instead rely on LLM judges [16].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The Chatbot Arena (Zheng et al., 2023) can be viewed in a similar light-as an effort to scale human preference judgments into a systematic ranking methodology, producing orderings that frequently diverge from those derived from standard benchmarks.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large-scale systems experience shows that production failure modes are subtle, and validation must include representative workloads and fault patterns [164], [165].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [165]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-4-Turbo holds the zero-shot state-of-the-art on numerous NLP tasks as of this writing, and achieves first place in human evaluations of chat-based models in Chatbot Arena [77].”
From Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings (2024) · cites this paper as [77]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“models may systematically assign overly favorable ratings to their own outputs-a phenomenon known as self-bias-which can distort evaluations of true model performance.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While other biases can be artificially introduced via perturbations of a completion, this is not the case for self-bias, making its study particularly challenging [Zheng et al., 2023].”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution (Zheng et al., 2023).”
From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Published LoCoMo numbers are not directly comparable because evaluation frameworks differ along multiple axes (Zheng et al., 2023; Wang et al., 2024): (i) answer LLM (gpt-4o-mini vs. gpt-4.1-mini), (ii) judge model and prompt (binary J-score vs. 7-step chainof-thought rubric), (iii) answer prompt (direct vs. CoT), and (iv) dataset split (LoCoMo-10 vs. LoCoMo-5).”
From SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Open-ended correctness is determined by majority vote from a three-way cross-vendor judge ensemble (Zheng et al. 2023); judge-produced auxiliary scores are used for preference alignment, bias identification, and memory fidelity.”
From FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents (2026) · cites this paper as Zheng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.”
From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large Language Models (LLMs) are gaining widespread acceptance as the gold standard for evaluation in numerous applications, thanks to their efficiency and significant reductions in cost & time compared to human evaluators (Kim et al., 2023”
From Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We use GPT-4o as an external judge. The judge evaluates Factual Consistency against the ground truth, penalizing contradictions or 'I don't know' responses if the answer exists [31].”
From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models have shown they can approximate the depth and consistency of manual human evaluation, but also provide better consistency and stability.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“as LLM-based evaluation is more effective than exact matches or F1 scores for numeric evaluation and long-form responses [14, 21, 68].”
From Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching. arXiv. (2025) · cites this paper as [68]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“previous work has shown that ‘LLM-as-a-judge’ typically has high success at capturing human preferences across a range of tasks40.”
From Training large language models on narrow tasks can lead to broad misalignment (2026) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MT-Bench and Chatbot Arena made LLM-as-a-judge a measurable methodology, quantifying both the ~80% judge-human agreement that justifies the paradigm and the position, verbosity, and self-enhancement biases that qualify it [131]”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [131]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the LLM-as-a-judge approach for Relevant Content and Fact Check evaluations, despite calibration through human review, may retain biases inherent to the judge model, including position bias and self-enhancement effects (Wang et al., 2024; Zheng et al., 2023).”
From Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our further experiments suggest that LLM judges (Li et al., 2023c;Zheng et al., 2023) provide superior accuracy in the evaluation of open-ended tasks.”
From L-Eval: Instituting Standardized Evaluation for Long Context Language Models (2023) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the Arena Elo ratings of LLMs (Zheng et al., 2023), which are indicative of their overall capabilities.”
From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-as-judge30 uses a second, ideally different-vendor model to evaluate the first against a rubric; it scales where manual checking cannot but imports its own failure modes.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A fixed LLM Judge (Liu et al. 2023;Zheng et al. 2023) additionally evaluates the paper, predicted critique, and generated suggestion for relevance, specificity, actionability, and overall quality on a 1-5 scale, where higher scores are better.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Zheng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In these settings, accurately judging response quality hinges on identifying whether a response is correct, which itself can require complex reasoning and domain knowledge (Tan et al., 2025;Zheng et al., 2023).”
From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Zheng et al. [2023] identify systematic biases in LLM judges (position preference, verbosity bias, self-enhancement) but study response quality evaluation, not citation-specific rubric tasks.”
From Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-asjudges-while correlating well with human judgment for simple question-answer tasks [154]-may be falsely lenient on more openended tasks, like slightly incorrect product descriptions (e.g., Coke Zero versus Diet Coke) [122].”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [154]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Judges were drawn from three vendors so that judge-family effects were measurable (Zheng et al., 2023;Panickssery et al., 2024), and all judges ran as nonreasoning models (Appendix E).”
From HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite its advantages, LLM-as-a-judge has limitations such as self-preference bias, where an LLM tends to favour its own responses; verbosity bias, where some LLMs may prefer longer, more detailed responses (Zheng et al., 2023;Stureborg et al., 2024); or sensitivity to prompt phrasing (Verga et al., 2024).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent advancements have demonstrated high alignment between”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, despite its advanced capabilities (Zheng et al., 2023), GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance.”
From Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, other studies show that LLM evaluators may be biased toward longer and more fluent answers, even when those answers are not necessarily more accurate or pedagogically better [35].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Zheng et al. [2023] showed that careful rubric prompting can match human agreement rates above 80%.”
From AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“accuracy, agreement, win rate, F 1 , or κ are used for binary or pairwise verdicts (e.g., Zheng et al., 2023;Verga et al., 2024;Lee et al., 2026)”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their results show GPT-4 is significantly less prone to this attack with below 10% success rate, while GPT-3.5 and Claude-v1 both suffer over 90% success rate.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prompts are taken from the library introduced by Zheng et al. (2023), all from the”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We also tried to incorporate Chain-of-Thought reasoning (Wei et al., 2022) in our prompt, but found that providing rationales along with the examples is more effective.”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The LLM-as-Judge paradigm was established by Zheng et al. (2023) (MT-Bench, Chatbot Arena), demonstrating that strong models such as GPT-4 can serve as reliable proxies for human annotators.”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vicuna-7/13/33B [8, 59] is a collection of open-source chatbot aligned from LLaMA with user-shared conversations.”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [59]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We use blind rubric-based adjudication, which is inspired by LLM-as-a-judge work that validates model-based outputs against human judgments [56], [57].”
From Guiding Human Validation of LLM-Generated Code via Verifiable Literate Programming (2026) · cites this paper as [56]
The cited source says what the claim attributes to it.
In the citing paper
“Although LLM benchmarks for general natural language processing (NLP) tasks exist20,21, they do not evaluate performance on relevant clinical tasks.”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“To reduce the cost of human evaluation, studies have adopted powerful LLMs as surrogate judges [338-339], an approach known as LLM-as-judge .”
From A Survey of Large Language Models (2026) · cites this paper as [338]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, recent work [338] has explored fine-tuning open-source LLMs as evaluators using human-annotated scoring data.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
The cited source says what the claim attributes to it.
In the citing paper
“Conversely, when tasked with evaluating options presented simultaneously, they can more reliably identify the superior response (Hu et al., 2024; Zheng et al., 2023).”
From Challenging the Evaluator: LLM Sycophancy Under User Rebuttal (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“MT-Bench and related work demonstrate that structured LLM-as-judge protocols can provide scalable model comparison when prompt templates and score aggregation are controlled [13].”
From HierMem: Context Curation Over Context Scaling — Hierarchical Memory with Invariant Constraint Placement for Long-Horizon LLM Conversations (2026) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“However, despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges [16] without validation on their credibility in reviewing scientific research.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Although modest in size, our dataset aligns with recent trends toward compact, high-quality benchmarks: MT-Bench (80 items) [16], GPQA-D (198 items) [4], AIME 2024/2025 (30 items each) [35], USAMO 2025 (6 items) [36] and PaperBench (20 items) [37].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Automatic judging LLMs have previously been proposed to act as judges to evaluate submissions for tasks (Zheng et al., 2023; Chen et al., 2024; Fu et al., 2023).”
From PaperBench: Evaluating AI's Ability to Replicate AI Research (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Findings from these researches indicate that LLM can mimic human behavior and provide evaluations that correspond with human judgments, revealing a potentially scalable and transparent alternative to costly and laborious human evaluations.”
From ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. (2023) demonstrated that strong LLMs achieve over 80% agreement with expert human ratings using MT-Bench and ChatBot Arena, positioning this method as a scalable and reliable alternative to manual annotation.”
From Multi-Agent debate system based on large language models: structured deliberation and validation in satellite communications (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLM judges are shown to exhibit systematic biases that favor completions with certain superficial characteristics rather than reflecting genuine quality differences.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Although this method may suggest self-bias, the authors do not dive into the statistical assumptions required in order to make a statistical inference on the presence and magnitude of the self-bias, e.g., how the two sets of scores are related.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent research25 has shown that strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, which is the same level of agreement between humans”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Prior research on LLM-as-a-Judge systems [50] has demonstrated that the input order of candidate responses can influence the evaluation outcome.”
From LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge (2025) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) have recently demonstrated remarkable performance across various tasks, leading to their widespread use as both the annotators (Peng et al., 2023;Xu et al., 2023) and evaluators (Zheng et al., 2023;Peng et al., 2023;Sun et al., 2023;Zhou et al., 2023;Gao et al., 2023;Wang et al., 2023b;Dubois et al., 2023;Wang et al., 2023a).”
From Large Language Models are not Fair Evaluators (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Pair-wise evaluation can give more accurate evaluation results than point-wise evaluation”
From Large Language Models are not Fair Evaluators (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“By introducing LLM as a judge (Zheng et al., 2023), this challenge can be addressed to some extent, as the LLM-as-judge evaluation prompt can effectively incorporate human preferences to adaptively evaluate open-ended tasks.”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Our verifier audit and trajectory analysis use a model to adjudicate agent trajectories, following the LLM-as-a-judge paradigm introduced by Zheng et al. [2023].”
From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies have explored the effectiveness of LLMs as evaluators and have reported strong correlations with human evaluations (Dubois et al., 2023;Zheng et al., 2023).”
From Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For all the strategies evaluation prompts P eval are adapted from Zheng et al. (2023); Zeng et al. (2023); Hada et al. (2023).”
From Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Many prior studies have explored this paradigm in text-only settings, analyzing agreement with human preferences and identifying sources of bias in LLM-based evaluation (Zheng et al., 2023; Chiang et al., 2024).”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Minimum viable alternatives , BERTScore, LLM-as-judge, entailment-based verification , are available [16,17] , but are not yet standard practice in the benchmarks and leaderboards that drive architectural decisions.”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. (2023) identify position bias, where models favor responses that appear earlier in the prompt, and verbosity bias, where longer responses are rated higher regardless of quality.”
From Sycophancy Negatively Affects LLM-as-a-Judge in Conflict Evaluation (2026) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Drawing from recent work showing the effectiveness of LLMs as evaluators, particularly GPT-4 (Zheng et al., 2023; Zeng et al., 2024), we adopt an LLM-as-a-Judge evaluation protocol for REALINSTRUCT.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, GPT-4's relatively high constraint-level accuracy could indicate scoring bias, as previous studies suggest GPT-4 tends to favor its own outputs (Zheng et al., 2023; Panickssery et al., 2024; Verga et al., 2024).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as previous studies suggest GPT-4 tends to favor its own outputs (Zheng et al., 2023; Panickssery et al., 2024; Verga et al., 2024).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We adapt the prompt proposed by Zheng et al. (2023), converting it to the Prometheus format, and introduce the option for tied responses, a feature not originally supported by Prometheus' pairwise ranking system.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“These systems hold their evaluation criterion fixed outside the loop, exposing them to benchmark saturation [12], judge biases [15, 16], and reward hacking [17].”
From The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators (2026) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“For model evaluation, we leverage GPT-4 as a proxy of the user to judge whether the system successfully handles the query (Zheng et al., 2023; Fu et al., 2023; Wang et al., 2023b).”
From EcoAssistant: Using LLM Assistant More Affordably and Accurately (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The LLM-as-a-Judge approach [125] leverages the reasoning capabilities of LLMs to evaluate agent responses based on qualitative criteria, assessing responses according to the criteria provided through instructions.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [125]
The cited source says what the claim attributes to it.
In the citing paper
“The use of LLMs as automated evaluators has gained significant attention as an alternative to costly human annotation (Zheng et al., 2023).”
From Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLM evaluators have been reported to favor more detailed and lengthy answers (Zheng et al., 2023).”
From L-Eval: Instituting Standardized Evaluation for Long Context Language Models (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Similar to Zheng et al. (2023), we found strong biases with LLM judges.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Order bias has been extensively studied (Jung et al., 2019;Wang et al., 2023a;Zheng et al., 2023), and it is well-known that language models can be influenced by the ordering of the responses in their evaluations.”
From Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Rubric-conditioned scoring in G-Eval and pairwise comparison in MT-Bench demonstrated that LLM judgments can align substantially with human preferences on open-ended generation tasks(Liu et al., 2023;Zheng et al., 2023).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The Prometheus-2 model is an LLM-as-a-judge model (Zheng et al., 2023) fine-tuned on feedback and judgment data generated by GPT-4 on a large set of custom score rubrics.”
From PeerQA: A Scientific Question Answering Dataset from Peer Reviews (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“establishing strong correlations with human judgement on general tasks [Zheng et al., 2023, Kim et al., 2023, 2024] while documenting systematic biases such as position, verbosity, and self-preference effects [Ye et al., 2024, Wataoka et al., 2024, Dubois et al., 2024, Kim et al., 2025, Ye et al., 2025].”
From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We use the LLM-asJudge evaluation protocol [Zheng et al., 2023] to assess answer correctness.”
From Accurate and Efficient Long-Term Memory for LLM Agents (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The complexity is far beyond what conventional LLM-as-a-Judge [45] methods are used for.”
From Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“As good-performing LLMs tend to perform better in LLM-as-a-judge tasks (Zheng et al., 2023)”
From DR-Arena: an Automated Evaluation Framework for Deep Research Agents (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Following Zheng et al. (2023), we run each pair twice, reversing the ordering of the references so as to account for any positional bias the judge model may have (e.g., always favoring the first model's response).”
From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown that GPT-4 accurately simulates human preferences in both summarisation (Rafailov et al., 2023) and instruction following (Dubois et al., 2023;Zheng et al., 2023), so we use it as the main performance metric in both tasks.”
From Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This method has shown strong correla-tion with human judgments across tasks.”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We observed positional bias in some weak LLM judges on Chatbot Arena where the first response was consistently preferred, similar to (Zheng et al., 2023).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“These benchmarks evaluate capabilities in writing, reasoning, extraction, and domain knowledge (e.g., STEM and humanities/social sciences), but their focus remains distinct from our objective.”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLM-as-a-judge systems offer a scalable alternative to expert annotation by substantially reducing cost and latency (Zheng et al., 2023).”
From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. [2023] introduced MT-Bench and Chatbot Arena, showing that GPT-4 judgments align with human preferences at scale.”
From AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. (2023) evaluate LLMs with other LLMs as a judge and show that GPT-4 has a high human alignment and agrees with humans on over 80% of evaluations.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“While previous studies have explored the concept of verbosity bias, they have tended to focus on specific cases.Zheng et al. (2023) limits their problem setting to questions answered with lists in their experiment on verbosity bias, and Huang et al. (2023) conducted experiments on summarization tasks.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Alongside their experiment on position bias, they experimented on verbosity bias by attempting a”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“there is also work demonstrating bias in using LLMs as evaluators (Wang et al., 2024;Zheng et al., 2023).”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Static benchmarks with clear ground-truth MMLU [29], GSM8k [11], and others like it do not capture flexible and interactive tasks needed to navigate scenarios in the real-world”
From Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents (2024) · cites this paper as [74]
The cited source says what the claim attributes to it.
In the citing paper
“Despite these challenges, judge validation continues to rely heavily on exact match.Established judge benchmarks-MT-Bench (Zheng et al., 2023), RewardBench (Lambert et al., 2025), JudgeBench (Tan et al., 2025)-and recent successors (Malik et al., 2026;Whitehouse et al., 2025;Jiang et al., 2025) privilege raw agreement as the headline validation metric.”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Opendialogue evaluations like MT-Bench Zheng et al. (2023) and Alpaca-Eval Li et al. (2023a), rely on subjective human or model-based judgments, making their outcomes susceptible to the influence of dialogue structure, and thereby, potentially compromising assessment validity.”
From Is Your LLM Outdated? A Deep Look at Temporal Generalization (2025) · cites this paper as Zheng et al. (2023)
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Since many of the existing datasets contain completions generated by relatively weak LLMs [Zhang et al., 2024, Zheng et al., 2023], we collect new model completions and corresponding LLM-as-a-judge judgments, which we will publicly release along with the publication.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Generally, we observe a positive correlation between the valid rate and the Arena rating of different LLM agents.”
From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“A separate line of work removes judge bias at the source via per-judge calibration on a labeled validation slice (Zheng et al., 2023).”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredIn the citing paper
“However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings.”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [56]
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Traditional evaluation metrics based on the similarity between outputs and reference answers (e.g., ROUGE [25], BLEU [32]) are also ineffective for these questions.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 6×
“5, Claude-V1, Vicuna-13B, Alpaca-13B [38], and LLaMA-13B [39].”
BackgroundInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
MethodQLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“There has been a proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [27, 2, 26, 7, 43].”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 1,693 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
MethodSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
MethodSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Research works printed but not linked: 2
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437-455, Boston, MA, April 2023. USENIX Association.
Cited by
Sorted by citations
Items 501–600 of 2,065
IOAgent: Democratizing Trustworthy HPC I/O Performance Diagnosis Capability via LLMs
Egersdoerfer, C., Sareen, A., Bez, J. L., Byna, S., Xu, D., & Dai, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00036 · Cited by 5
The silicon service spectrum: warmth and competence explain people's preferences for AI assistants
Hernandez, I. A., & Chekili, A. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2024.1396533 · Cited by 6
LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites
Sollenberger, Z., Patel, J., Munley, C., Jarmusch, A., & Chandrasekaran, S. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00238 · Cited by 5
TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models
Shao, W., Lei, M., Hu, Y., Gao, P., Xu, P., Zhang, K., Meng, F., Huang, S., Li, H., Qiao, Y., et al. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536930 · Cited by 33
Fernández-Becerra, L., González-Santamarta, M. Á., Guerrero‐Higueras, Á. M., Rodríguez-Lera, F. J., & Olivera, V. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4773675 · Cited by 10
Agent-as-a-Judge: Evaluate Agents with Agents
Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10934 · Cited by 5
PaperBench: Evaluating AI's Ability to Replicate AI Research
Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.01848 · Cited by 310
“to act as judges to evaluate submissions for tasks (Zheng et al., 2023; Chen et al., 2024; Fu et al., 2023).”
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Liu, X., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., & Chu, X. (2025) · arXiv · doi:10.48550/arxiv.2502.00299 · Cited by 4
“We use NIAH [26] to evaluate the long-context retrieval capability of LLMs. NIAH assesses how well LLM extracts hidden tricked information from extensive documents, and following LLM-as-a-Judge [43] we apply GPT-4o-mini [44] to assess the accuracy of the retrieved information.”
Self-citationChallenging the Evaluator: LLM Sycophancy Under User Rebuttal
Kim, S. W., & Khashabi, D. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1222 · Cited by 6
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.335 · Cited by 4
Evaluating Text Summarization Techniques and Factual Consistency with Language Models
Islam, M. M., Muhammad, U., & Oussalah, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826032 · Cited by 6
DocSpider: a dataset of cross-domain natural language querying for MongoDB
Özer, A. G., Çekinel, R. F., Toroslu, İ. H., & Karagöz, P. (2025) · Natural language processing. · doi:10.1017/nlp.2024.63 · Cited by 6
BegoniaGPT: Cultivating the large language model to be an exceptional K-12 English teacher
Rong, L., Zhang, Y., Tiwari, P., & Mei, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107488 · Cited by 4
Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants
Richards, J., & Wessel, M. S. (2025) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse67031.2025.00009 · Cited by 4
Kunuku, M. T., & Dehbozorgi, N. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99261-2_49 · Cited by 4
Applying Large Language Model For Relevance Search In Tencent
Ye, D., Liu, J., Hu, J., Fan, J., Tian, B., Liang, H., & Ma, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737193 · Cited by 7
Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Malaviya, C., Chang, J. C., Roth, D., Iyyer, M., Yatskar, M., & Lo, K. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.24 · Cited by 26
Wang, M., Chen, Y., Huang, X., & Lai, Y. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100568 · Cited by 8
Cheng, Z., Lai, L., Liu, Y., Cheng, K., & Qi, X. (2026) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet65726.2026.11632928 · Cited by 26
“Several limitations warrant discussion: Evaluation Methodology: The LLM-as-judge approach, while cost-effective and scalable, may introduce biases such as preferring verbose answers or penalizing valid alternative phrasings [14].”
XMutant: XAI-based fuzzing for deep learning systems
Chen, X., Biagiola, M., Riccio, V., d’Amorim, M., & Stocco, A. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10792-1 · Cited by 8
“Large language models like Chat-GPT, on the other hand, provide a viable automated solution for validation and are increasingly regarded as reliable as humans for comprehending and explaining small chunks of text, such as those produced by XMutant for IMDB, as highlighted by recent studies [95,107].”
An empirical study of testing practices in open source AI agent frameworks and agentic applications
Hasan, M. M., Li, H., Fallahzadeh, E., Rajbahadur, G. K., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10857-9 · Cited by 25
Empirical Phenomenology: Action as Disclosure and the Science of Opaque Public Systems
Sharks, L. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20326136 · Cited by 4
Gordetzki, P., Blohm, I., Clegg, M., Schakols, F., & Hofstetter, R. (2026) · Information Systems Research · doi:10.1287/isre.2024.0952 · Cited by 7
LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias
Kucia, F. J., Chakraborty, A., & Wróblewska, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29918-5_38 · Cited by 8
A Neuro-Symbolic Approach to Extracurricular Activity Recommendation Based on Cognitive Profiling
Bukina, T. V., Khramova, M. V., Smirnov, N. M., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29744-0_21 · Cited by 4
Jailbreaking large language models: models, origins, and evolution of attacks and defenses
Li, X. H., Wu, J., Zheng, Q., Wang, H., Fan, M. Y., Hu, S., Guo, J., & Ting, L. I. U. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0196 · Cited by 4
Watermarking Datasets for LLM Fine-tuning
Qiu, J., Yang, X., Li, S., Chen, K., Zhang, W., & Yu, N. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889741 · Cited by 4
Liu, S., Shi, L., Shi, Y., Gao, Y., & Sun, X. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113210 · Cited by 7
Malo in the Code Jungle: Explainable Fault Localization for Decentralized Applications
Zhang, H., Wu, J., Wu, Z., Chen, Z., Lin, D., Chen, J., Zhou, Y., & Zheng, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3578816 · Cited by 4
Self-citationSurvey and improvement strategies for gene prioritization with large language models
Neeley, M. B., Qi, G., Wang, G., Tang, R., Mao, D., Liu, C., Pasupuleti, S., Yuan, B., Xia, F., Liu, P., et al. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf148 · Cited by 9
Extension decisions in open source software ecosystem
Onagh, E., & Nayebi, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112552 · Cited by 6
Lee, J.-H., & Rew, J. (2025) · Applied Sciences · doi:10.3390/app15179775 · Cited by 9
LLM-SAP: Large Language Models Situational Awareness-Based Planning
Wang, L., & Zhong, H. (2024) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew63481.2024.10645429 · Cited by 7
Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
Li, R., Wu, Y., & He, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01275 · Cited by 5
Palaniappan, S., Mali, R., Cuomo, L., Vitale, M., Youssef, A., Madathil, A. P., Murugesan, M., Bettini, A., Magistris, G. D., & Veneri, G. (2024) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/222032-ms · Cited by 4
Jayashankar, P., Roy, T., Chattopadhyay, S., Arshad, M., & Sarkar, S. (2024) · Journal of Business and Industrial Marketing · doi:10.1108/jbim-10-2024-0811 · Cited by 4
A Comprehensive Review of AI Advancement Using testFAILS and testFAILS-2 for the Pursuit of AGI
Kumar, Y., Lin, M., Paredes, C., Li, D., Yang, G., Kruger, D., Li, J. J., & Morreale, P. A. (2024) · Electronics · doi:10.3390/electronics13244991 · Cited by 6
Self-citationFreeman, B. S., Arriola, K., Cottell, D., Lawlor, E., Erdman, M., Sutherland, T., & Wells, B. (2024) · American Journal of Artificial Intelligence · doi:10.11648/j.ajai.20240802.16 · Cited by 6
LLM Augmentations to support Analytical Reasoning over Multiple Documents
Yousuf, R. B., Defelice, N., Sharma, M., Xu, S., & Ramakrishnan, N. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826114 · Cited by 6
Audio-Language Datasets of Scenes and Events: A Survey
Wijngaard, G., Formisano, E., Esposito, M., & Dumontier, M. (2025) · IEEE Access · doi:10.1109/access.2025.3534621 · Cited by 7
Evaluating simulated teaching audio for teacher trainees using RAG and local LLMs
Fang, K., Tang, C., & Jing, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-87898-5 · Cited by 6
Knowledge Distillation and Transformer-Based Framework for Automatic Spine CT Report Generation
Batool, H., Mukhtar, A., Khawaja, S. G., Alghamdi, N. S., Khan, A. M., Qayyum, A., Adil, R., Khan, Z., Akram, M. U., Akbar, M. U., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3546131 · Cited by 5
QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture
Prakash, S., Cheng, A., Yik, J., Tschand, A., Ghosal, R., Uchendu, I., Quaye, J. A., Ma, J., Grampurohit, S., Giannuzzi, S., et al. (2025) · IEEE Computer Architecture Letters · doi:10.1109/lca.2025.3541961 · Cited by 4
Deng, Y., Qiu, X., Tan, X., Qu, C., Pan, J., Cheng, Y., Xu, Y., & Chu, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887899 · Cited by 6
Chen, P., & Li, Z. (2025) · Electronics · doi:10.3390/electronics14081662 · Cited by 4
Self-citationAgent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-Agent LLMs
Li, X., Su, L., Jia, P., Cheng, S., Wang, J., Yin, D., & Zhao, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3749099 · Cited by 39
MailEx: Email Event and Argument Extraction
Srivastava, S., Singh, G., Matsumoto, S., Raz, A., Costa, P., Poore, J., & Yao, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.801 · Cited by 5
ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation
Wang, C., Zhou, H., Hu, Y., Huo, Y., Li, B., Liu, T., Xiao, T., & Zhu, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29878 · Cited by 9
Yamanaka, J., & 隆, 城. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31254 · Cited by 4
Generalizing Verifiable Instruction Following
Pyatkin, V., Malik, S., Graf, V., Ivison, H., Huang, S., Dasigi, P., Lambert, N., & Hajishirzi, H. (2025) · arXiv · doi:10.48550/arxiv.2507.02833 · Cited by 4
Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
Fan, Z., Wang, Y., Polisetty, S., & Fung, Y. R. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1037 · Cited by 4
Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models
Lyu, C., Wu, M., & Aji, A. F. (2024) · arXiv · doi:10.48550/arxiv.2402.13887 · Cited by 4
Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting
Wang, R., Wang, H., Mi, F., Xue, B., Chen, Y., Wong, K.-F., & Xu, R. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.299 · Cited by 4
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
“The proprietary models include GPT-4 ( gpt-4-0314 ), GPT-4-Turbo ( gpt-4-turbo-2024-04-09 ), and GPT-3.”
InfluentialAgentic Context Engineering: Evolving Contexts for Self-Improving Language Models
Zhang, Q., Hu, C., Upasani, S., Ma, B., Hong, F., Kamanuru, V., Rainton, J., Wu, C., Ji, M., Li, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.04618 · Cited by 370
User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal
Liu, Y., Zhang, M. J. Q., & Choi, E. (2025) · arXiv · doi:10.48550/arxiv.2507.23158 · Cited by 3
“One hypothesis explaining these unintuitive re-sults is that distilling on the more targeted data from neg improves performance on the easier tasks in MT-Bench, but not on the much harder tasks in WildBench.”
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7
“To enhance the capability of LLMs to model long texts, several studies extend the context window through position interpolation (kaiokendev, 2023; Chen et al., 2023; Peng et al., 2023), and continuing to fine-tune the models on multi-turn conversations and long text datasets (Zheng et al., 2023).”
MethodKIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions
Xu, F., Lo, K., Soldaini, L., Kuehl, B., Choi, E., & Wadden, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.770 · Cited by 3
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 493
Free-MAD: Consensus-Free Multi-Agent Debate
Cui, Y., Fu, H., Zhang, H., Wang, L., & Zuo, C. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1600 · Cited by 28
Self-citationPeerQA: A Scientific Question Answering Dataset from Peer Reviews
Baumgärtner, T., Briscoe, T., & Gurevych, I. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.22 · Cited by 3
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3
Vision Language Modeling of Content, Distortion and Appearance for Image Quality Assessment
Zhou, F., Gu, T., Huang, Z., & Qiu, G. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3516392 · Cited by 5
Du, Z., & Hashimoto, K. (2025) · Information · doi:10.3390/info16040254 · Cited by 5
Don't Get Too Excited - Eliciting Emotions in LLMs
Fazzi, G. F., Hinge, J. S., Heinrich, S., & Burelli, P. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720191 · Cited by 12
Leveraging LLMs and RAG for Enhanced Football Talent Scouting
Martire, F. A., & Ragazzi, D. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-94931-9_24 · Cited by 4
DialRet: Enhancing Dialogue Retention for Multi-session Conversations
Na, Y., Kim, D., & Chae, D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8186-0_7 · Cited by 2
Janusko, T., Bochnia, R., Harnisch, M., Krauß, A.-M., Richter, D., Hempel, G., Tomschke, S., Anke, J., & Thiele, M. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96235-6_30 · Cited by 3
Selective fine-tuning for large language models via matrix nuclear norm
Xia, T., Li, Y., Wu, Y., & Chang, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104259 · Cited by 3
Question-Answer Extraction from Scientific Articles Using Knowledge Graphs and Large Language Models
Azarbonyad, H., Zhu, Z. L., Cheirmpos, G., Afzal, Z., Yadav, V., & Tsatsaronis, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730068 · Cited by 3
Yang, L., Luo, Y., Gao, H., Fan, Y., Zhang, J., Li, X., Dong, X., Gu, B., Jin, Z. G., & Yang, M. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728560 · Cited by 5
Gender inclusive language generation framework: A reasoning approach with RAG and CoT
Chinnan, S. P. M., Drury‐Grogan, M. L., & Chakravarthi, B. R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114092 · Cited by 4
Amato, F., Conte, C., Fonisto, M., Moccardi, A., & Moscato, F. (2025) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-96099-4_9 · Cited by 3
FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance
Zhang, M., Fu, J., Warrier, T., Wang, Y., Tan, T., & Huang, K. (2025) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3768292.3770433 · Cited by 7
From algorithms to systems: integrating computation into drug discovery
Bradley, A. R., Rossall, A., Pairaudeau, G., & Deane, C. M. (2025) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2025.2601102 · Cited by 4
UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
Xiong, Z., Ye, X., Yaman, B., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37252-9_21 · Cited by 3
Kim, S., Choi, Y., Kim, B., Oh, J. H., & Yoo, H. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408453 · Cited by 3
The Sorting Function: Mediation, Predation, and the Foreclosed Question
Sharks, L. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20308546 · Cited by 3
AI Realtor: Towards Grounded Persuasive Language Generation for Automated Copywriting
Wu, J., Yang, C., Wu, Y., Mahns, S., Wang, C., Zhu, H., Fang, F., & Xu, H. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813172 · Cited by 4
“To evaluate the effectiveness of listing descriptions generated by different models, we draw inspiration from ChatArena (Zheng et al., 2023) and conduct an online survey to collect pairwise human feedback comparing different models’ outputs.”
Riaz, H., Haroon, A., Baig, M., Rizwan, M. D., Bajwa, M. N., & Fraz, M. M. (2026) · International Conference on Robotics and Automation in Industry (ICRAI) · doi:10.1109/icrai70912.2026.11551956 · Cited by 3
“In addition, we also employ an LLM-as-a-Judge framework [20] to assess semantic quality across Coverage, Consistency, Accuracy, Style, and Conciseness.”
Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement
Jin, H., & Chen, H. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00638-5 · Cited by 13
Pumpo, M. D., Villani, L., Gualano, M. R., Buonsenso, D., Raffaelli, F., Donà, D., Laurenti, P., Maio, V., Boccia, S., & Ricciardi, W. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1874389 · Cited by 3
Accelerating Contextualization in AI Large Language Models Using Vector Databases
Tareaf, R. B., AbuJarour, M., Engelman, T., Liermann, P., & Klotz, J. (2024) · International Conference on Information Networking · doi:10.1109/icoin59985.2024.10572088 · Cited by 3
AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations
Hartman, J., Sagtani, H., Tibshirani, J., & Mehrotra, R. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688060 · Cited by 3
Can we trust LLMs as relevance judges?
Bencke, L. R., Paula, F. S. F., dos Santos, B. G. T., & Moreira, V. P. (2024) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2024.243130 · Cited by 3
Enhance Large Language Models for Scientific Paper Question Answering
Chen, Z., Jiang, Y., Yang, Y. T., Jiang, X., & Zhang, Q. (2024) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml62565.2024.10779743 · Cited by 3
Shayaninasab, M., Zahoor, M., & Yalçın, Ö. N. (2024) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3652988.3673968 · Cited by 3
Dolomites : Domain-Specific Long-Form Methodical Tasks
Malaviya, C., Agrawal, P., Ganchev, K., Srinivasan, P., Huot, F., Berant, J., Yatskar, M., Das, D., Lapata, M., & Alberti, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00727 · Cited by 8
RadChat: A Radiology Chatbot Incorporating Clinical Context for Radiological Reports Summarization
Mei, X., Yang, L., Gao, D., Cai, X., Liu, T., & Han, J. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822010 · Cited by 3
Advancing Risk and Quality Assurance: A RAG Chatbot for Improved Regulatory Compliance
Hillebrand, L., Berger, A., Uedelhoven, D., Berghaus, D., Warning, U., Dilmaghani, T., Kliem, B., Schmid, T., Loitz, R., & Sifa, R. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825431 · Cited by 3
Human-In-The-Loop Reasoning for Traffic Sign Detection: Collaborative Approach YOLO with Video-LLaVA
Azarafza, M., Idrees, F., Bejnordi, A. E., Steinmetz, C., Henkler, S., & Rettberg, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-84457-7_9 · Cited by 3
Enhancing Large Language Model Inference Efficiency via Lookahead Cache Filtering
Ou, J., Chen, Y., Jiang, S., & Tian, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888646 · Cited by 3
Enhancing Reasoning Capacity of SLM Using Cognitive Enhancement
Pan, J., Wong, S. L., Chia, X. W., & Yuan, Y. (2025) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic64266.2025.10920811 · Cited by 4
BiTA: Bi-directional tuning for lossless acceleration in large language models
Lin, F., Yi, H., Yang, Y., Li, H., Yu, X., Lu, G., & Xiao, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127305 · Cited by 3
Summary report auto-generation based on hierarchical corpus using large language model
Guan, X., Ye, Z., Cao, B., & Fan, J. (2025) · Displays · doi:10.1016/j.displa.2025.103055 · Cited by 3
Diverse AI Feedback For Large Language Model Alignment
Yu, T., Lin, T.-E., Wu, Y., Yang, M., Huang, F., & Li, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00746 · Cited by 12
Harnessing Large Language Models for Curated Code Reviews
Sghaier, O. B., Weyssow, M., & Sahraoui, H. A. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00039 · Cited by 5
ScamFerret: Detecting Scam Websites Autonomously with Large Language Models
Nakano, H., Koide, T., & Chiba, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97620-9_1 · Cited by 8
Ivy: A Hybrid Knowledge-Based and Generative AI Coach for Explaining Procedural Skills
Dass, R. K., Madhusudhana, R. H., Deye, E. C., Verma, S., Bydlon, T., Brazil, G., & Goel, A. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_17 · Cited by 5
Metadata
- Authors
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 46595-46623
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.05685
- OpenAlex
- W7133239970
- Cited by
- 1,888
- References
- 2
- Citations by year
- 2026: 1115 · 2025: 413 · 2024: 308 · 2023: 51
- FWCI
- 2548.93
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- VMware
- Samsung
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, LLM-as-a-judge, automated assessment, MT-Bench, Chatbot Arena, benchmarking, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
