The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index; 4 of the 29 sources its claims cite were checked for support.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Open accessAbstract
Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper examines whether strong LLMs can serve as judges for chatbot responses by comparing their judgments with human preferences on two new benchmarks, MT-bench and Chatbot Arena. It finds that GPT-4 can reach over 80% agreement with human preferences, comparable to human-human agreement, while also documenting several judge biases and mitigation strategies.
Key findings
- GPT-4’s judgments aligned closely with human experts on MT-bench: 85% agreement without ties, versus 81% agreement among humans; on Chatbot Arena, GPT-4 also showed high agreement with crowd votes, above 80%.
- Position bias was substantial: with the default prompt, consistency was 65.0% for GPT-4, 46.2% for GPT-3.5, and 23.8% for Claude-v1; swapping positions and few-shot examples raised GPT-4 consistency to 77.5%.
- Verbosity bias appeared in a “repetitive list” attack: Claude-v1 and GPT-3.5 failed 91.3% of cases, while GPT-4 failed 8.7%.
- Reference-guided grading reduced math-judging failures from 14/20 under the default prompt to 3/20, with chain-of-thought at 6/20; self-enhancement was suggested by GPT-4 favoring itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate, but the study could not confirm the bias.
Methods
- The authors created MT-bench, 80 multi-turn questions across 8 categories with 10 questions each, and Chatbot Arena, a crowdsourced anonymous battle platform; they collected about 3K expert votes from 58 labelers on MT-bench and about 30K Arena votes, sampling 3K for analysis.
- They compared LLM judges—mainly GPT-4, GPT-3.5, and Claude-v1—using pairwise comparison, single-answer grading, and reference-guided grading against human expert and crowd votes, measuring agreement and win rates.
- They also evaluated LLaMA and Vicuna variants on MMLU, TruthfulQA, and MT-bench with GPT-4 single-answer grading to test how preference-based and capability-based benchmarks complement each other.
Limitations
- The paper states that it emphasizes helpfulness and largely neglects safety, honesty, and harmlessness; it also notes that helpfulness dimensions such as accuracy, relevance, and creativity are combined into one metric, and its bias mitigations are preliminary.
- The paper reports that self-enhancement bias could not be determined because of limited data and small differences, and that its position-bias test is difficult because the compared answers are very similar and sometimes indistinguishable even to humans.
- The text makes evident that few-shot judging may introduce new biases and makes API calls about 4× more expensive, while zero-shot Vicuna-13B as a judge had low consistency (11.2–16.2%) and high error rates (22.5–78.8%), limiting its use as a scalable open-source judge.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
56 Own result · 35 Own interpretation · 7 Sources not found · 7 No basis given · 3 Partly supported · 2 Own method · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Once aligned with humans, these chat models are strongly preferred by human users over the original, unaligned models on which they are built.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites [31] source
- Partly supportedcites [2] sourceExperimental — accuracy not yet measured
Section 4.5 (p. 21) says crowdworkers clearly preferred the online RLHF models in comparisons with context-distilled and earlier RLHF models. But those preference comparisons do not establish that users strongly preferred them over the original, unaligned base models; the paper discusses base-model improvements on other evaluations (Introduction, p. 4). (LLM, conf=0.86)
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [48]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [14]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that some of the biases are minor or can be mitigated.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that in math/coding category, GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 is still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.3 Win rates under different judges
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, a small high-quality conversation dataset can quickly teach the model a style preferred by GPT-4 (or approximately human) but cannot improve MMLU significantly, as shown by the Vicuna-7B (selected) which is trained with only 4.8M tokens or 3K conversations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most LLM judges favor the first position.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only GPT-4 successfully detected this attack.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As in Table 2, we found all of them exhibit strong position bias.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, it has a high error rate (from 22.5% to 78.8%) because of its weaker instruction-following capability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude-v1 favors itself with a 25% higher win rate.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 12 (Appendix), the few-shot judge can significantly increase the consistency of GPT-4 from 65.0% to 77.5%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plaindirectn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 22
- It may state two separate claims as one. 29
- The quote only partly states this claim. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As LLMs continue to improve, they show potential in replacing human annotators in many tasks [17,20].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3 LLM as a Judge
What the sources say
- Supportedcites [17] source
The Discussion (p. 4) says LLMs could transform text annotation across a variety of tasks and that ChatGPT may already outperform crowd annotators. The Introduction (p. 2) also describes the possibility of annotating entire samples with ChatGPT. These passages support the hedged claim about potential replacement in annotation tasks, particularly those performed by crowd-workers. (LLM, conf=0.88)
- Partly supportedcites [20] sourceExperimental — accuracy not yet measured
The Discussion and Conclusion (Section 5; page unavailable) says ChatGPT showed great potential as a data annotation tool for implicit hateful tweets. It does not establish that LLMs could replace human annotators across many tasks; it also urges caution about incorrect decisions. (LLM, conf=0.86)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
What the sources say
- Partly supportedcites [21] sourceExperimental — accuracy not yet measured
The Introduction (p. 1) presents Dynabench as dynamic data collection with humans and models in the loop, in contrast to static benchmarking. Caveats and Objections (p. 8) explicitly says it anticipates dataset saturation. The copy does not identify overfitting as a challenge of static benchmarks; its discussion of annotators ‘overfitting’ on models (p. 7) describes a potential risk of the dynamic approach instead. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the base LLaMA models showing competitive performance on conventional benchmarks (Table 8), its answers to open-ended questions are often not preferred by humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
- Also stated in: § GPT-4 Judgment: · plain
“Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement ( §4.2, Table 4).”
- Also stated in: § 4.2 High agreement between GPT-4 and humans · plain
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § GPT-4 Judgment: · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The consistency improves significantly from 16.2% to 65.0%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 56.8% when including all three labels, and 85.5% when excluding tie predictions and labels, significantly outperforming random guesses of 33% and 50% respectively, and show positive signals to match GPT-4 (66% and 87% respectively).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite GPT-4 knows how to solve the question (if asked separately), it made a wrong judgement saying both assistants' wrong answers are correct.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 14:
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the classification formulation, every output is recognizable (error rate 0%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 15, Vicuna-13B has a consistency rate from 11.2% to 16.2% across different prompt templates, much lower than all the closed-sourced models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Figure 13 (Appendix), we present an example of an elementary math question in which GPT-4 makes an incorrect judgment.It's worth noting that although GPT-4 can solve the problem (when asked separately), it was misled by the provided answers, ultimately resulting in incorrect judgment.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 3 shows the failure rate of LLM judges under this attack, demonstrating that all LLMs may be prone to verbosity bias though GPT-4 defends significantly better than others.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 11, the position bias is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLMs can achieve an agreement rate of over 80%, on par with the level of agreement among human experts, establishing a foundation for an LLM-based evaluation framework.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 7 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing GPT-4 and other LLM judges, we find they reach a similar non-tie agreement ratio between humans but the number of non-tied votes from GPT-4 is much larger.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that fine-tuning on high-quality dialog datasets (i.e., ShareGPT) can consistently improve the model performance on MMLU and the improvement scales with fine-tuning data size.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-3.5 and Claude-v1 show a verbosity bias towards the longer and repetitive answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both tables, GPT-4 with single-answer grading matches both pairwise GPT-4 and human preferences very well.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On math and coding questions, LLM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the longer prompts make API calls 4× more expensive.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, even with the CoT prompt, we find that in many cases LLM makes exactly the same mistake as the given answers in its problem-solving process (See example in Figure 15 (Appendix), suggesting that LLM judge may still be misled by the context.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar pattern from other LLM judges such as Claude/GPT-3.5.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the pairwise comparison may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically;”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks -benchmarks like MMLU [19] and HELM [24] cannot effectively tell the difference between these aligned models and the base models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not matchedcites [19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [24]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This misalignment of conventional benchmarks underscores the core problem driving this paper: the need for a robust and scalable automated method to evaluate LLM alignment with human preferences.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“single answer grading may be unable to discern subtle differences between specific pairs, and its results may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Conversational benchmarks, like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.However, the diversity and complexity of their questions often fall short in challenging the capabilities of the latest chatbots.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 2.1 Motivation
What the sources say
- Not matchedcites [35]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, high consistency may not imply high accuracy and we are not sure whether the few-shot examples will introduce new biases.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that GPT-4 is more affirmative and less suffered from position bias but other models also perform well when they give an affirmative answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“LLMs are known to have limited math and reasoning capability [10], which results in its failure of grading such questions because they do not know the correct answers.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They focus on different aspects of a model and the recommended way is to comprehensively evaluate models with both kinds of benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To invite participants, we obtained their consent by letting them sign an application form.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.1 MT-bench human evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another more aggressive approach is to assign positions randomly, which can be effective at a large scale with the correct expectations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 8, no single benchmark can determine model quality, meaning that a comprehensive evaluation is needed.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This bias is not unique to our context and has been seen in human decision-making [3,34] and other ML domains [22,41].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matchedcites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [34]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [41]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Chain-of-thought is a widely used technique to improve LLM's reasoning capability [47].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
What the sources say
- Not matchedcites [47]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our LLM-as-a-judge approach can automate and scale platforms of this nature.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that using LLM-as-a-judge to approximate human preferences is highly feasible and could become a new standard in future benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Users are required to accept the terms of use, which obtain their consent and give us the right to release the conversation data.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.2 Chatbot Arena
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These responses are rendered as natural languages and are difficult to be parsed automatically, making the model less useful in a scalable and automatic evaluation pipeline.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-as-a-judge offers two key benefits: scalability and explainability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.2 Advantages of LLM-as-a-Judge
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that this test is challenging because the answers are very similar and occasionally indistinguishable even to humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The single-answer grading can be converted into pairwise comparison results for calculating the agreement.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Honesty and harmlessness are crucial for a chat assistant as well [2].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 6 Discussion
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The paper relies on this without support.
LLMs trained with RLHF already exhibit strong human alignment.
“Because these models are often trained with RLHF, they already exhibit strong human alignment.”
The paper relies on this without support.
There has not been a systematic study of the LLM-as-a-judge approach.
“However, there has not been a systematic study of this approach.”
The paper relies on this without support.
Human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.
“While largely overlooked by existing LLM benchmarks, human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.”
The paper relies on this without support.
The human labelers are mostly graduate students and are considered experts and more skilled than average crowd workers.
“The labelers are mostly graduate students so they are considered experts and more skilled than average crowd workers.”
The paper relies on this without support.
Human evaluation is the gold standard for assessing human preferences.
“While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly.”
The paper relies on this without support.
Model services such as GPT-4 can become expensive with a growing number of evaluations.
“However, model services such as GPT-4 can also become expensive with a growing number of evaluations.”
The paper relies on this without support.
Popular open-sourced LLMs, e.g. Vicuna-13B, show strong language understanding capability and are much cheaper than close-sourced LLMs.
“On the other hand, popular open-sourced LLMs, e.g.Vicuna-13B shows strong language understanding capability, and are much cheaper than close-sourced LLMs.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“To mitigate stochastic variance and”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Zheng et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Others employ the LLM-as-a-judge methodology [36], yet these efforts lack both a comprehensive framework design and verification of human consistency.”
From DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) · cites this paper as [36]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“LLM-based evaluators are mainly used in well-defined tasks that require simple information extraction and classification, which has been shown to have high precision Zheng et al. (2023).”
From TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (2024) · cites this paper as Zheng et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The choice of judge can shift accuracy by 5-10 percentage points on identical retrieval output (Zheng et al., 2023).”
From SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additional biases further complicate assessment due to inherent limitations in LLMs: length bias (favoring unnecessarily long outputs), position bias (order effects in candidates), and style bias (favoring certain linguistic styles over content) [337-338,343], potentially distorting results regardless of model capability.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While this method improves scalability and efficiency, it may also inherit biases present in the training data, such as position and verbosity bias.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While automated and scalable, evaluation results can be affected by several factors [337-338], including prompt sensitivity, demonstration setup, and data contamination.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“modern LLMs exhibit diverse generation behaviors in terms of verbosity, reasoning style, and response length.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although the 3-vote majority consensus protocol reduces stochastic fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family (Zheng et al., 2023;Ye et al., 2024), and may be more lenient on verbose near-misses than a human annotator would be (Zheng et al., 2023;Ye et al., 2024;Chen et al., 2024).”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The LLM-as-a-judge framework using GPT-5-mini as evaluator (Zheng et al., 2023) introduces its own sources of variance.”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges [16] without validation on their credibility in reviewing scientific research.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, concurrent works often omit a rigorous backward pass or 'verification' and instead rely on LLM judges [16].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The Chatbot Arena (Zheng et al., 2023) can be viewed in a similar light-as an effort to scale human preference judgments into a systematic ranking methodology, producing orderings that frequently diverge from those derived from standard benchmarks.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large-scale systems experience shows that production failure modes are subtle, and validation must include representative workloads and fault patterns [164], [165].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [165]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-4-Turbo holds the zero-shot state-of-the-art on numerous NLP tasks as of this writing, and achieves first place in human evaluations of chat-based models in Chatbot Arena [77].”
From Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings (2024) · cites this paper as [77]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“models may systematically assign overly favorable ratings to their own outputs-a phenomenon known as self-bias-which can distort evaluations of true model performance.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While other biases can be artificially introduced via perturbations of a completion, this is not the case for self-bias, making its study particularly challenging [Zheng et al., 2023].”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution (Zheng et al., 2023).”
From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Published LoCoMo numbers are not directly comparable because evaluation frameworks differ along multiple axes (Zheng et al., 2023; Wang et al., 2024): (i) answer LLM (gpt-4o-mini vs. gpt-4.1-mini), (ii) judge model and prompt (binary J-score vs. 7-step chainof-thought rubric), (iii) answer prompt (direct vs. CoT), and (iv) dataset split (LoCoMo-10 vs. LoCoMo-5).”
From SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Open-ended correctness is determined by majority vote from a three-way cross-vendor judge ensemble (Zheng et al. 2023); judge-produced auxiliary scores are used for preference alignment, bias identification, and memory fidelity.”
From FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents (2026) · cites this paper as Zheng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.”
From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large Language Models (LLMs) are gaining widespread acceptance as the gold standard for evaluation in numerous applications, thanks to their efficiency and significant reductions in cost & time compared to human evaluators (Kim et al., 2023”
From Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We use GPT-4o as an external judge. The judge evaluates Factual Consistency against the ground truth, penalizing contradictions or 'I don't know' responses if the answer exists [31].”
From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models have shown they can approximate the depth and consistency of manual human evaluation, but also provide better consistency and stability.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“as LLM-based evaluation is more effective than exact matches or F1 scores for numeric evaluation and long-form responses [14, 21, 68].”
From Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching. arXiv. (2025) · cites this paper as [68]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“previous work has shown that ‘LLM-as-a-judge’ typically has high success at capturing human preferences across a range of tasks40.”
From Training large language models on narrow tasks can lead to broad misalignment (2026) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MT-Bench and Chatbot Arena made LLM-as-a-judge a measurable methodology, quantifying both the ~80% judge-human agreement that justifies the paradigm and the position, verbosity, and self-enhancement biases that qualify it [131]”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [131]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the LLM-as-a-judge approach for Relevant Content and Fact Check evaluations, despite calibration through human review, may retain biases inherent to the judge model, including position bias and self-enhancement effects (Wang et al., 2024; Zheng et al., 2023).”
From Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our further experiments suggest that LLM judges (Li et al., 2023c;Zheng et al., 2023) provide superior accuracy in the evaluation of open-ended tasks.”
From L-Eval: Instituting Standardized Evaluation for Long Context Language Models (2023) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the Arena Elo ratings of LLMs (Zheng et al., 2023), which are indicative of their overall capabilities.”
From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-as-judge30 uses a second, ideally different-vendor model to evaluate the first against a rubric; it scales where manual checking cannot but imports its own failure modes.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A fixed LLM Judge (Liu et al. 2023;Zheng et al. 2023) additionally evaluates the paper, predicted critique, and generated suggestion for relevance, specificity, actionability, and overall quality on a 1-5 scale, where higher scores are better.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Zheng et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In these settings, accurately judging response quality hinges on identifying whether a response is correct, which itself can require complex reasoning and domain knowledge (Tan et al., 2025;Zheng et al., 2023).”
From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Zheng et al. [2023] identify systematic biases in LLM judges (position preference, verbosity bias, self-enhancement) but study response quality evaluation, not citation-specific rubric tasks.”
From Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-asjudges-while correlating well with human judgment for simple question-answer tasks [154]-may be falsely lenient on more openended tasks, like slightly incorrect product descriptions (e.g., Coke Zero versus Diet Coke) [122].”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [154]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Judges were drawn from three vendors so that judge-family effects were measurable (Zheng et al., 2023;Panickssery et al., 2024), and all judges ran as nonreasoning models (Appendix E).”
From HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite its advantages, LLM-as-a-judge has limitations such as self-preference bias, where an LLM tends to favour its own responses; verbosity bias, where some LLMs may prefer longer, more detailed responses (Zheng et al., 2023;Stureborg et al., 2024); or sensitivity to prompt phrasing (Verga et al., 2024).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent advancements have demonstrated high alignment between”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, despite its advanced capabilities (Zheng et al., 2023), GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance.”
From Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, other studies show that LLM evaluators may be biased toward longer and more fluent answers, even when those answers are not necessarily more accurate or pedagogically better [35].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Zheng et al. [2023] showed that careful rubric prompting can match human agreement rates above 80%.”
From AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“accuracy, agreement, win rate, F 1 , or κ are used for binary or pairwise verdicts (e.g., Zheng et al., 2023;Verga et al., 2024;Lee et al., 2026)”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their results show GPT-4 is significantly less prone to this attack with below 10% success rate, while GPT-3.5 and Claude-v1 both suffer over 90% success rate.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prompts are taken from the library introduced by Zheng et al. (2023), all from the”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We also tried to incorporate Chain-of-Thought reasoning (Wei et al., 2022) in our prompt, but found that providing rationales along with the examples is more effective.”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The LLM-as-Judge paradigm was established by Zheng et al. (2023) (MT-Bench, Chatbot Arena), demonstrating that strong models such as GPT-4 can serve as reliable proxies for human annotators.”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vicuna-7/13/33B [8, 59] is a collection of open-source chatbot aligned from LLaMA with user-shared conversations.”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [59]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We use blind rubric-based adjudication, which is inspired by LLM-as-a-judge work that validates model-based outputs against human judgments [56], [57].”
From Guiding Human Validation of LLM-Generated Code via Verifiable Literate Programming (2026) · cites this paper as [56]
The cited source says what the claim attributes to it.
In the citing paper
“Although LLM benchmarks for general natural language processing (NLP) tasks exist20,21, they do not evaluate performance on relevant clinical tasks.”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“To reduce the cost of human evaluation, studies have adopted powerful LLMs as surrogate judges [338-339], an approach known as LLM-as-judge .”
From A Survey of Large Language Models (2026) · cites this paper as [338]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, recent work [338] has explored fine-tuning open-source LLMs as evaluators using human-annotated scoring data.”
From A Survey of Large Language Models (2026) · cites this paper as [338]
The cited source says what the claim attributes to it.
In the citing paper
“Conversely, when tasked with evaluating options presented simultaneously, they can more reliably identify the superior response (Hu et al., 2024; Zheng et al., 2023).”
From Challenging the Evaluator: LLM Sycophancy Under User Rebuttal (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“MT-Bench and related work demonstrate that structured LLM-as-judge protocols can provide scalable model comparison when prompt templates and score aggregation are controlled [13].”
From HierMem: Context Curation Over Context Scaling — Hierarchical Memory with Invariant Constraint Placement for Long-Horizon LLM Conversations (2026) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“However, despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges [16] without validation on their credibility in reviewing scientific research.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Although modest in size, our dataset aligns with recent trends toward compact, high-quality benchmarks: MT-Bench (80 items) [16], GPQA-D (198 items) [4], AIME 2024/2025 (30 items each) [35], USAMO 2025 (6 items) [36] and PaperBench (20 items) [37].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Automatic judging LLMs have previously been proposed to act as judges to evaluate submissions for tasks (Zheng et al., 2023; Chen et al., 2024; Fu et al., 2023).”
From PaperBench: Evaluating AI's Ability to Replicate AI Research (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Findings from these researches indicate that LLM can mimic human behavior and provide evaluations that correspond with human judgments, revealing a potentially scalable and transparent alternative to costly and laborious human evaluations.”
From ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. (2023) demonstrated that strong LLMs achieve over 80% agreement with expert human ratings using MT-Bench and ChatBot Arena, positioning this method as a scalable and reliable alternative to manual annotation.”
From Multi-Agent debate system based on large language models: structured deliberation and validation in satellite communications (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLM judges are shown to exhibit systematic biases that favor completions with certain superficial characteristics rather than reflecting genuine quality differences.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Although this method may suggest self-bias, the authors do not dive into the statistical assumptions required in order to make a statistical inference on the presence and magnitude of the self-bias, e.g., how the two sets of scores are related.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent research25 has shown that strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, which is the same level of agreement between humans”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Prior research on LLM-as-a-Judge systems [50] has demonstrated that the input order of candidate responses can influence the evaluation outcome.”
From LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge (2025) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) have recently demonstrated remarkable performance across various tasks, leading to their widespread use as both the annotators (Peng et al., 2023;Xu et al., 2023) and evaluators (Zheng et al., 2023;Peng et al., 2023;Sun et al., 2023;Zhou et al., 2023;Gao et al., 2023;Wang et al., 2023b;Dubois et al., 2023;Wang et al., 2023a).”
From Large Language Models are not Fair Evaluators (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Pair-wise evaluation can give more accurate evaluation results than point-wise evaluation”
From Large Language Models are not Fair Evaluators (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“By introducing LLM as a judge (Zheng et al., 2023), this challenge can be addressed to some extent, as the LLM-as-judge evaluation prompt can effectively incorporate human preferences to adaptively evaluate open-ended tasks.”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Our verifier audit and trajectory analysis use a model to adjudicate agent trajectories, following the LLM-as-a-judge paradigm introduced by Zheng et al. [2023].”
From DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies have explored the effectiveness of LLMs as evaluators and have reported strong correlations with human evaluations (Dubois et al., 2023;Zheng et al., 2023).”
From Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“For all the strategies evaluation prompts P eval are adapted from Zheng et al. (2023); Zeng et al. (2023); Hada et al. (2023).”
From Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Many prior studies have explored this paradigm in text-only settings, analyzing agreement with human preferences and identifying sources of bias in LLM-based evaluation (Zheng et al., 2023; Chiang et al., 2024).”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Minimum viable alternatives , BERTScore, LLM-as-judge, entailment-based verification , are available [16,17] , but are not yet standard practice in the benchmarks and leaderboards that drive architectural decisions.”
From When Context Overwhelms: A Study of Long-Context vs. Retrieval-Based QA Under Noise (2026) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. (2023) identify position bias, where models favor responses that appear earlier in the prompt, and verbosity bias, where longer responses are rated higher regardless of quality.”
From Sycophancy Negatively Affects LLM-as-a-Judge in Conflict Evaluation (2026) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Drawing from recent work showing the effectiveness of LLMs as evaluators, particularly GPT-4 (Zheng et al., 2023; Zeng et al., 2024), we adopt an LLM-as-a-Judge evaluation protocol for REALINSTRUCT.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, GPT-4's relatively high constraint-level accuracy could indicate scoring bias, as previous studies suggest GPT-4 tends to favor its own outputs (Zheng et al., 2023; Panickssery et al., 2024; Verga et al., 2024).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as previous studies suggest GPT-4 tends to favor its own outputs (Zheng et al., 2023; Panickssery et al., 2024; Verga et al., 2024).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We adapt the prompt proposed by Zheng et al. (2023), converting it to the Prometheus format, and introduce the option for tied responses, a feature not originally supported by Prometheus' pairwise ranking system.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“These systems hold their evaluation criterion fixed outside the loop, exposing them to benchmark saturation [12], judge biases [15, 16], and reward hacking [17].”
From The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators (2026) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“For model evaluation, we leverage GPT-4 as a proxy of the user to judge whether the system successfully handles the query (Zheng et al., 2023; Fu et al., 2023; Wang et al., 2023b).”
From EcoAssistant: Using LLM Assistant More Affordably and Accurately (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The LLM-as-a-Judge approach [125] leverages the reasoning capabilities of LLMs to evaluate agent responses based on qualitative criteria, assessing responses according to the criteria provided through instructions.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [125]
The cited source says what the claim attributes to it.
In the citing paper
“The use of LLMs as automated evaluators has gained significant attention as an alternative to costly human annotation (Zheng et al., 2023).”
From Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLM evaluators have been reported to favor more detailed and lengthy answers (Zheng et al., 2023).”
From L-Eval: Instituting Standardized Evaluation for Long Context Language Models (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Similar to Zheng et al. (2023), we found strong biases with LLM judges.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Order bias has been extensively studied (Jung et al., 2019;Wang et al., 2023a;Zheng et al., 2023), and it is well-known that language models can be influenced by the ordering of the responses in their evaluations.”
From Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Rubric-conditioned scoring in G-Eval and pairwise comparison in MT-Bench demonstrated that LLM judgments can align substantially with human preferences on open-ended generation tasks(Liu et al., 2023;Zheng et al., 2023).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The Prometheus-2 model is an LLM-as-a-judge model (Zheng et al., 2023) fine-tuned on feedback and judgment data generated by GPT-4 on a large set of custom score rubrics.”
From PeerQA: A Scientific Question Answering Dataset from Peer Reviews (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“establishing strong correlations with human judgement on general tasks [Zheng et al., 2023, Kim et al., 2023, 2024] while documenting systematic biases such as position, verbosity, and self-preference effects [Ye et al., 2024, Wataoka et al., 2024, Dubois et al., 2024, Kim et al., 2025, Ye et al., 2025].”
From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We use the LLM-asJudge evaluation protocol [Zheng et al., 2023] to assess answer correctness.”
From Accurate and Efficient Long-Term Memory for LLM Agents (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The complexity is far beyond what conventional LLM-as-a-Judge [45] methods are used for.”
From Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“As good-performing LLMs tend to perform better in LLM-as-a-judge tasks (Zheng et al., 2023)”
From DR-Arena: an Automated Evaluation Framework for Deep Research Agents (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Following Zheng et al. (2023), we run each pair twice, reversing the ordering of the references so as to account for any positional bias the judge model may have (e.g., always favoring the first model's response).”
From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Previous work has shown that GPT-4 accurately simulates human preferences in both summarisation (Rafailov et al., 2023) and instruction following (Dubois et al., 2023;Zheng et al., 2023), so we use it as the main performance metric in both tasks.”
From Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This method has shown strong correla-tion with human judgments across tasks.”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“We observed positional bias in some weak LLM judges on Chatbot Arena where the first response was consistently preferred, similar to (Zheng et al., 2023).”
From SkillAggregation: Reference-free LLM-Dependent Aggregation (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“These benchmarks evaluate capabilities in writing, reasoning, extraction, and domain knowledge (e.g., STEM and humanities/social sciences), but their focus remains distinct from our objective.”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“LLM-as-a-judge systems offer a scalable alternative to expert annotation by substantially reducing cost and latency (Zheng et al., 2023).”
From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. [2023] introduced MT-Bench and Chatbot Arena, showing that GPT-4 judgments align with human preferences at scale.”
From AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Zheng et al. (2023) evaluate LLMs with other LLMs as a judge and show that GPT-4 has a high human alignment and agrees with humans on over 80% of evaluations.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“While previous studies have explored the concept of verbosity bias, they have tended to focus on specific cases.Zheng et al. (2023) limits their problem setting to questions answered with lists in their experiment on verbosity bias, and Huang et al. (2023) conducted experiments on summarization tasks.”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Alongside their experiment on position bias, they experimented on verbosity bias by attempting a”
From Verbosity Bias in Preference Labeling by Large Language Models (2023) · cites this paper as Zheng et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“there is also work demonstrating bias in using LLMs as evaluators (Wang et al., 2024;Zheng et al., 2023).”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Static benchmarks with clear ground-truth MMLU [29], GSM8k [11], and others like it do not capture flexible and interactive tasks needed to navigate scenarios in the real-world”
From Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents (2024) · cites this paper as [74]
The cited source says what the claim attributes to it.
In the citing paper
“Despite these challenges, judge validation continues to rely heavily on exact match.Established judge benchmarks-MT-Bench (Zheng et al., 2023), RewardBench (Lambert et al., 2025), JudgeBench (Tan et al., 2025)-and recent successors (Malik et al., 2026;Whitehouse et al., 2025;Jiang et al., 2025) privilege raw agreement as the headline validation metric.”
From Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (2026) · cites this paper as Zheng et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Opendialogue evaluations like MT-Bench Zheng et al. (2023) and Alpaca-Eval Li et al. (2023a), rely on subjective human or model-based judgments, making their outcomes susceptible to the influence of dialogue structure, and thereby, potentially compromising assessment validity.”
From Is Your LLM Outdated? A Deep Look at Temporal Generalization (2025) · cites this paper as Zheng et al. (2023)
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Since many of the existing datasets contain completions generated by relatively weak LLMs [Zhang et al., 2024, Zheng et al., 2023], we collect new model completions and corresponding LLM-as-a-judge judgments, which we will publicly release along with the publication.”
From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Generally, we observe a positive correlation between the valid rate and the Arena rating of different LLM agents.”
From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“A separate line of work removes judge bias at the source via per-judge calibration on a labeled validation slice (Zheng et al., 2023).”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Zheng et al., 2023
Experimental — accuracy not yet measuredIn the citing paper
“However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings.”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [56]
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Traditional evaluation metrics based on the similarity between outputs and reference answers (e.g., ROUGE [25], BLEU [32]) are also ineffective for these questions.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 6×
“5, Claude-V1, Vicuna-13B, Alpaca-13B [38], and LLaMA-13B [39].”
BackgroundInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
MethodQLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“There has been a proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [27, 2, 26, 7, 43].”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 1,693 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
MethodSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
MethodSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Research works printed but not linked: 2
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437-455, Boston, MA, April 2023. USENIX Association.
Cited by
Sorted by citations
Items 201–300 of 2,066
Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns
Bian, N., Lin, H., Liu, P., Lu, Y., Zhang, C.-P., He, B., Han, X., & Sun, L. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3476030 · Cited by 60
LLM based QA chatbot builder: A generative AI-based chatbot builder for question answering
Salim, M. S., Hossain, S. I., Jalal, T., Bose, D. K., & Basher, M. J. I. (2024) · SoftwareX · doi:10.1016/j.softx.2024.102029 · Cited by 30
Mindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17066 · Cited by 24
Maximising Large Language Model Utility in Cardiovascular Care: A Practical Guide
Nolin-Lapalme, A., Thériault-Lauzier, P., Corbin, D., Tastet, O., Sharma, A., Hussin, J., Kadoury, S., Jiang, R., Krahn, A. D., Gallo, R., et al. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.05.024 · Cited by 29
LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J., Reif, E., Kallarackal, K., Chang, M., Terry, M. A., & Dixon, L. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456354 · Cited by 34
Deng, P., Zhou, W., & Wu, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890620 · Cited by 23
Huang, W., Zhou, G., Lapata, M., Vougiouklis, P., Montella, S., & Pan, J. Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113648 · Cited by 29
Query Performance Prediction Using Relevance Judgments Generated by Large Language Models
Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., & de Rijke, M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3736402 · Cited by 61
HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning
Yang, Y., Kim, J., Kim, Y., Ho, N., Thorne, J. P., & Yun, S.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.365 · Cited by 47
Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data
Wu, Z., Weng, Z., Peng, W., Yang, X., Li, A., Davis, L. S., & Jiang, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3357503 · Cited by 27
Self-citationVeCLIP: Improving CLIP Training via Visual-Enriched Captions
Lai, Z., Zhang, H., Zhang, B., Wu, W., Bai, H., Timofeev, A., Du, X., Gan, Z., Shan, J., Chuah, C., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_7 · Cited by 64
Self-citationHumanELY: Human evaluation of LLM yield, using a novel web-based evaluation toolPreprint
Awasthi, R., Mishra, S. V., Mahapatra, D., Khanna, A. K., Maheshwari, K. K., Cywiński, J. B., Papay, F., & Mathur, P. (2023) · medRxiv · doi:10.1101/2023.12.22.23300458 · Cited by 23
Guthrie, E., Levy, D., & del Carmen, G. (2024) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2024.02.016 · Cited by 22
SECURE: Benchmarking Large Language Models for Cybersecurity
Bhusal, D., Alam, T., Nguyen, L., Mahara, A., Lightcap, Z., Frazier, R., Fieblinger, R., Torales, G. L., Blakely, B., & Rastogi, N. (2024) · Annual Computer Security Applications Conference · doi:10.1109/acsac63791.2024.00019 · Cited by 50
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 22
3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
Xiong, H., Zhuge, Y., Zhu, J., Zhang, L., & Lu, H. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557620 · Cited by 34
Exploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 42
Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models
Zhang, J., Wu, Q., Xu, Y., Cao, C. J., Du, Z., & Psounis, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30178 · Cited by 39
Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 56
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00737 · Cited by 21
Retrieval-style In-context Learning for Few-shot Hierarchical Text Classification
Chen, H., Zhao, Y. H., Chen, Z., Wang, M., Li, L., Zhang, M., & Zhang, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00697 · Cited by 23
A Brief Review on Benchmarking for Large Language Models Evaluation in Healthcare
Cilar, L., Chen, H., Chen, A., Topaz, M., Tam, W., Bian, J., & Štiglic, G. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70010 · Cited by 27
Liu, S., Liu, S., Sha, L., Zeng, Z., Gašević, D., & Liu, Z. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3570775 · Cited by 24
“I Understand Why I Got This Grade”: Automatic Short Answer Grading (ASAG) with Feedback
Aggarwal, D., Sil, P., Raman, B., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98420-4_22 · Cited by 21
BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning
Liu, R., Li, C., Ge, Y., Li, T. H., Shan, Y., & Li, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01296 · Cited by 54
LLMs4OM: Matching Ontologies with Large Language Models
Giglou, H. B., D’Souza, J., Engel, F., & Auer, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78952-6_3 · Cited by 62
LM-Polygraph: Uncertainty estimation for language models
Fadeeva, E., Vashurin, R., Tsvigun, A., Vazhentsev, A., Petrakov, S., Fedyanin, K., Vasilev, D., Goncharova, E., Panchenko, A., Panov, M., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.41 · Cited by 20
Banerjee, A., Satish, A., & Wörndl, W. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-87654-7_3 · Cited by 32
Automated Evaluation of Large Vision-Language Models on Self-Driving Corner Cases
Chen, K., Li, Y., Zhang, W., Liu, Y., Li, P., Gao, R., Hong, L., Meng, T., Zhao, X., Li, Z., et al. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00759 · Cited by 68
Self-citationAggarwal, T., Salatino, A. A., Osborne, F., & Motta, E. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104262 · Cited by 34
LLM-based Solutions for Healthcare Chatbots: a Comparative Analysis
Montagna, S., Aguzzi, G., Ferretti, S., Pengo, M. F., Klopfenstein, L. C., Ungolo, M., & Magnini, M. (2024) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops59983.2024.10503257 · Cited by 21
A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot
Du, F., Ma, X.-J., Yang, J., Liu, Y., Luo, C.-R., Wang, X., Jiang, H.-O., & Jing, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3767-3 · Cited by 25
So you want your private LLM at home? A survey and benchmark of methods for efficient GPTs
Tuggener, L., Sager, P., Taoudi-Benchekroun, Y., Grewe, B. F., & Stadelmann, T. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00036 · Cited by 23
Federated Transfer Learning for On-Device LLMs Efficient Fine Tuning Optimization
Li, C., Gu, B., Zhao, Z., Qu, Y., Xin, G., Huo, J., & Gao, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020068 · Cited by 20
The Responsible Development of Automated Student Feedback with Generative AI
Lindsay, E. D., Zhang, M., Johri, A., & Bjerva, J. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016572 · Cited by 43
Li, Z., Muhtar, D., Gu, F., He, Y., Zhang, X., Xiao, P., He, G., & Zhu, X. X. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.06.003 · Cited by 41
Self-citationFoundation models and Transformers for anomaly detection: A survey
Ammar, M. B., Mendoza, A., Belkhir, N., Antoine Manzanera, B., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103517 · Cited by 25
Strengths and Weaknesses of LLM-Based and Rule-Based NLP Technologies and Their Potential Synergies
Karanikolas, N. Ν., Manga, E., Samaridi, N. E., Stergiopoulos, V., Tousidou, E., & Vassilakopoulos, M. G. (2025) · Electronics · doi:10.3390/electronics14153064 · Cited by 24
AgentCoord: Visually exploring coordination strategy for LLM-based multi-agent collaboration
Pan, B., Lu, J., Wang, K., Li, Z., Wen, Z., Feng, Y., Zhu, M., & Chen, W. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104338 · Cited by 27
Self-citationHackMentor: Fine-Tuning Large Language Models for Cybersecurity
Zhang, J., Wen, H., Deng, L., Xin, M., Li, Z., Li, L., Zhu, H., & Sun, L. (2023) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom60117.2023.00076 · Cited by 24
Self-citationDescribing Differences in Image Sets with Natural Language
Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., & Yeung, S. Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02284 · Cited by 32
Self-citationThe Promise and Peril of Generative AI: Evidence from GPT as Sell-Side AnalystsPreprint
Li, E. X., Shen, M., Tu, Z., & Zhou, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4480947 · Cited by 21
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
“As a result, the neutrality of the evaluator is in question, and the evaluation can suffer from biases where the LLM evaluators diverge from humans in systematic ways (Zheng et al., 2024; Bai et al., 2024).”
A Comprehensive Evaluation of Neural SPARQL Query Generation From Natural Language Questions
Diallo, P. A. K. K., Reyd, S., & Zouaq, A. (2024) · IEEE Access · doi:10.1109/access.2024.3453215 · Cited by 27
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 33
LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services
Lazuka, M., Anghel, A., & Parnell, T. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00022 · Cited by 31
MoE-L ightning : High-Throughput MoE Inference on Memory-constrained GPUs
Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., & Stoica, I. G. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707267 · Cited by 47
Self-citationLeveraging Large Language Models to Detect NPM Malicious Packages
Zahan, N., Burckhardt, P., Lysenko, M., Aboukhadijeh, F., & Williams, L. A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00146 · Cited by 45
Quality Assessment for AI Generated Images With Instruction Tuning
Wang, J., Duan, H., Zhai, G., & Min, X. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3651009 · Cited by 30
“The frozen LLM adopted in our model is Vicuna [85], a recently released decoder-only LLM finetuned from LLaMA [79].”
Mao, L., Long, R., Huang, H., Ma, L., & Zhou, S. (2026) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce69494.2026.11504499 · Cited by 21
Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation
Da, Y., Liu, X., Yin, F., Zhong, M., Bansal, H., Han, J., & Chang, K.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.245 · Cited by 41
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
Wang, W., Ren, Y., Luo, H., Li, T., Yan, C., Chen, Z., Wang, W., Li, Q., Lu, L., Zhu, X., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_27 · Cited by 64
A Survey on the Integration and Optimization of Large Language Models in Edge Computing Environments
Bhardwaj, S., Singh, P., & Pandit, M. K. (2024) · 2024 16th International Conference on Computer and Automation Engineering (ICCAE) · doi:10.1109/iccae59995.2024.10569285 · Cited by 18
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
“This is a set of inputs taken from a variety of open-source instruction following and dialogue training and evaluation datasets (Bai et al., 2022; Ganguli et al., 2022; Gudibande, 2023; Köpf et al., 2023; Wang et al., 2023; Zheng et al., 2023), curated by Dubois et al. (2023).”
BackgroundMethodLaboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 24
Williams, A. R., Burke-Moore, L., Chan, R. S.-Y., Enock, F. E., Nanni, F., Sippy, T., Chung, Y.-L., Gabašová, E., Hackenburg, K., & Bright, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0317421 · Cited by 44
FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models
Li, Y., Liu, X., Wang, X., Lee, B., Wang, S., Rocha, A. D. R., & Lin, W. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3597211 · Cited by 59
Compromising LLM Driven Embodied Agents With Contextual Backdoor Attacks
Liu, A., Zhou, Y., Liu, X., Zhang, T., Liang, S., Wang, J., Pu, Y., Li, T., Zhang, J., Zhou, W., et al. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3555410 · Cited by 28
GPT4All: An Ecosystem of Open Source Compressed Language Models
Anand, Y., Nussbaum, Z., Treat, A., Miller, A., Guo, R., Schmidt, B., Duderstadt, B., & Mulyar, A. (2023) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2023.nlposs-1.7 · Cited by 19
Zhou, Z., Yang, T., & Hu, K. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385748 · Cited by 17
Lee, S. Y.-T., Bahukhandi, A., Liu, D., & Ma, K. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456398 · Cited by 19
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Zhang, J., Wang, T., Zhang, H., Lu, P., & Zheng, F. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73113-6_12 · Cited by 17
Self-citationTheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.14161 · Cited by 17
Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approach
Sartori, C. C., Blum, C., Bistaffa, F., & Corominas, G. R. (2024) · IEEE Access · doi:10.1109/access.2024.3524176 · Cited by 19
Nawara, D., & Kashef, R. F. (2025) · IEEE Access · doi:10.1109/access.2025.3599832 · Cited by 21
LEXI: Large Language Models Experimentation Interface
Laban, G., Laban, T., & Güneş, H. (2024) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3687272.3688296 · Cited by 25
Towards Interpretable Radiology Report Generation via Concept Bottlenecks Using a Multi-agentic RAG
Alam, H., Srivastav, D., Kadir, M. A., & Sonntag, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_18 · Cited by 23
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., & Lewis, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18796 · Cited by 16
RouteLLM: Learning to Route LLMs with Preference Data
Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2406.18665 · Cited by 15
“Evaluation benchmarks : We evaluate our routers on three widely-used academic benchmarks: MMLU [16] consisting of 14,042 questions across 57 subjects, MT Bench [27] with 160 open-ended questions using LLM-as-a-judge, and GSM8K [11] with over 1,000 grade school math problems.”
Self-citationWhy Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
“Do not distribute.”
Self-citationLLM-as-a-Judge: automated evaluation of search query parsing using large language models
Baysan, M. S., Uysal, S., İşlek, İ., Karaman, Ç. Ç., & Güngör, T. (2025) · Frontiers in Big Data · doi:10.3389/fdata.2025.1611389 · Cited by 15
LLMs for legal reasoning: A unified framework and future perspectives
Nguyen, H.-T., Fungwacharakorn, W., Zin, M. M., Goebel, R. G., Toni, F., Stathis, K., & Satoh, K. (2025) · Computer law & security review · doi:10.1016/j.clsr.2025.106165 · Cited by 26
Zhang, C., D’Haro, L. F., Chen, Y., Zhang, M., & Li, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29923 · Cited by 31
Macanovic, A., & Przepiorka, W. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02381-9 · Cited by 20
Benchmarking PathCLIP for Pathology Image Analysis
Zheng, S., Cui, X., Sun, Y., Li, J., Li, H., Zhang, Y., Chen, P., Jing, X., Ye, Z., & Yang, L. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01128-4 · Cited by 15
Comparative Analysis of Open-Source Language Models in Summarizing Medical Text Data
Chen, Y., Wang, Z., & Zulkernine, F. (2024) · 2024 IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00030 · Cited by 15
EASTER: Learning to Split Transformers at the Edge Robustly
Guo, X., Jiang, Q., Shen, Y., Pimentel, A. D., & Stefanov, T. P. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3438995 · Cited by 15
Wang, Z., Duan, J., Yuan, C., Chen, Q., Chen, T., Zhang, Y., Wang, R., Shi, X., & Xu, K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109553 · Cited by 25
Priv-IQ: A Benchmark and Comparative Evaluation of Large Multimodal Models on Privacy Competencies
Shahriar, S., & Dara, R. A. (2025) · AI · doi:10.3390/ai6020029 · Cited by 20
TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy
Chen, Y., Liu, Q., Zhang, Y., Sun, W., Ma, X., Yang, W., Shi, D., Mao, J., & Yin, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714863 · Cited by 42
Using In-Context Learning to Improve Dialogue Safety
Meade, N., Gella, S., Hazarika, D., Gupta, P., Jin, D., Reddy, S., Liu, Y., & Hakkani-Tur, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.796 · Cited by 13
Verbosity Bias in Preference Labeling by Large Language Models
Saito, K., Wachi, A., Wataoka, K., & Akimoto, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.10076 · Cited by 15
VeriScore: Evaluating the factuality of verifiable claims in long-form text generation
Song, Y., Kim, Y., & Iyyer, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.552 · Cited by 15
Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents
Piatti, G., Jin, Z., Kleiman-Weiner, M., Schölkopf, B., Sachan, M., & Mihalcea, R. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3548 · Cited by 14
“Static benchmarks with clear ground-truth MMLU [29], GSM8k [11], and others like it do not capture flexible and interactive tasks needed to navigate scenarios in the real-world [22, 47, 73].”
Voting or Consensus? Decision-Making in Multi-Agent Debate
Kaesberg, L. B., Becker, J., Wahle, J. P., Ruas, T., & Gipp, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.606 · Cited by 55
“A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent’s solution (Zheng et al., 2023).”
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., Soboroff, I., Dang, H. T., & Lin, J. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730090 · Cited by 14
Zhang, H., Zhang, Y., Li, J., Liu, J., & Ji, L. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110157 · Cited by 16
Self-citationBoosting Jailbreak Attack with Momentum
Zhang, Y., & Wei, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888812 · Cited by 32
The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs
Gourabathina, A., Gerych, W., Pan, E., & Ghassemi, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732121 · Cited by 21
Variational autoencoder for design of synthetic viral vector serotypes
Lyu, S., Sowlati‐Hashjin, S., & Garton, M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00787-2 · Cited by 27
LLMEval: A Preliminary Study on How to Evaluate Large Language Models
Zhang, Y., Zhang, M., Yuan, H., Liu, S., Shi, Y., Gui, T., Zhang, Q., & Huang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29934 · Cited by 31
DriveLLaVA: Human-Level Behavior Decisions via Vision Language Model
Zhao, R., Yuan, Q., Li, J., Fan, Y., Li, Y., & Gao, F. (2024) · Sensors · doi:10.3390/s24134113 · Cited by 18
GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?
Taesiri, M. R., Feng, T., Bezemer, C., & Nguyen, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02118 · Cited by 15
InVERGe: Intelligent Visual Encoder for Bridging Modalities in Report Generation
Deria, A., Kumar, K., Chakraborty, S., Mahapatra, D., & Roy, S. S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00208 · Cited by 17
CoLE: A collaborative legal expert prompting framework for large language models in law
Li, B., Fan, S., Zhu, S., & Wen, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113052 · Cited by 15
A Causality-Aware Paradigm for Evaluating Creativity of Multimodal Large Language Models
Huang, Z., Zhong, S., Zhou, P., Gao, S., Žitnik, M., & Lin, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3539433 · Cited by 16
Tung, Y.-H., Yang, Z.-R., Shen, M.-W., Chang, C., Chen, C., & Ho, L.-C. (2025) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2025.105371 · Cited by 17
Toward Holistic Evaluation of Recommender Systems Powered by Generative Models
Deldjoo, Y., Mehta, N., Sathiamoorthy, M., Zhang, S., Castells, P., & McAuley, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730354 · Cited by 20
1.5 million materials narratives generated by chatbots
Park, Y. J., Jerng, S. E., Yoon, S., & Li, J. (2024) · Scientific Data · doi:10.1038/s41597-024-03886-w · Cited by 15
Ayanwale, M. A., Adelana, O. P., Bamiro, N. B., Olatunbosun, S. O., Idowu, K. O., & Adewale, K. A. (2025) · F1000Research · doi:10.12688/f1000research.161637.1 · Cited by 15
Metadata
- Authors
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 46595-46623
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.05685
- OpenAlex
- W7133239970
- Cited by
- 1,888
- References
- 2
- Citations by year
- 2026: 1115 · 2025: 413 · 2024: 308 · 2023: 51
- FWCI
- 2548.93
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- VMware
- Samsung
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, LLM-as-a-judge, automated assessment, MT-Bench, Chatbot Arena, benchmarking, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
