The DOI matches its DataCite record; 63 of the 63 references it cites were found in a registry or an index; 8 of the 13 sources its claims cite were checked for support.
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Open accessAbstract
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
ChatEval is a multi-agent debate framework for LLM-based text evaluation, using multiple LLM agents with diverse role prompts and different communication strategies to discuss and judge generated responses. Experiments on FairEval and Topical-Chat show improved alignment with human assessment over single-agent evaluation and several baselines.
Key findings
- On FairEval open-ended question answering, multi-agent ChatEval improved accuracy over single-agent evaluation by 6.2% for ChatGPT (53.8% → 60.0%) and 2.5% for GPT-4 (61.3% → 63.8%), surpassing FairEval’s best MEC+BPC results (ChatGPT 58.7%, GPT-4 62.5%). Human annotators averaged 71.7% accuracy, with individual annotators at 68.8%, 76.3%, and 70.0%.
- On Topical-Chat dialogue response generation, ChatEval with GPT-4 improved average Spearman and Kendall-Tau correlation by 0.096 (16.3%) and 0.057 (10.0%), respectively, over G-EVAL-4.
- Diverse role prompts were essential: with the same role prompt, multi-agent ChatGPT scored 53.8% accuracy and 0.25 Kappa, versus 60.0% and 0.33 with diverse role prompts; the same-role condition did not effectively improve over single-agent (53.8%, 0.27).
- For ChatGPT on FairEval, one-by-one communication was strongest among tested strategies (60.0% accuracy, 0.33 Kappa), compared with simultaneous-talk (55.0%, 0.28) and simultaneous-talk-with-summarizer (55.0%, 0.27). Accuracy peaked at 62.5% with 3–4 roles before declining at 5 roles, while increasing discussion turns showed no significant upward trend.
Methods
- ChatEval treats each LLM as a debater agent with a distinct role/persona and a shared prompt template; agents autonomously exchange responses over discussion turns without human intervention. Final judgments use majority vote for comparison formats or average score for direct-score formats. Three communication strategies are tested: one-by-one, simultaneous-talk, and simultaneous-talk-with-summarizer.
- Evaluated on two benchmarks: FairEval for open-ended question answering (80 questions, human annotations from Wu et al. 2023, three annotators comparing Vicuna-13B and ChatGPT) and Topical-Chat for dialogue response generation (60 dialogue contexts, responses from 6 systems, human annotations from Mehri & Eskenazi 2020; four dimensions: naturalness, coherence, engagingness, groundedness).
- Uses OpenAI GPT-family models GPT-4 and ChatGPT (GPT-3.5-turbo) with temperature 0, in homogeneous groups. Default configuration is one-by-one communication, 2 agents, and 2 discussion turns, with position calibration in both single-agent and multi-agent settings. Metrics are Accuracy and Kappa for FairEval, and turn-level Spearman and Kendall-Tau correlations for Topical-Chat.
Limitations
- The paper states that it focuses on homogeneous groups of LLMs, with heterogeneous groups left for future research; it also notes that communication strategies other than one-by-one did not perform as robustly and require further exploration and optimization.
- The paper states that continual discussion often leads to stagnation or even degradation of performance, possibly due to ever-increasing context length.
- Evident from the text: evaluation is limited to two benchmarks and GPT-family models, and the paper does not report testing with non-GPT or heterogeneous model groups.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Own result · 6 Own interpretation · 5 Partly supported · 3 Supported · 3 Own method · 1 Sources not found · 1 No full text of the sources · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, these methods have been shown to exhibit a relatively weak correlation with human judgments, particularly in the context of tasks involving open-ended generation or requiring domain-specific expertise (Novikova et al., 2017).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Partly supportedcites Novikova et al., 2017 sourceExperimental — accuracy not yet measured
Section 7.1 reports that no metric achieved even a moderate utterance-level correlation with human ratings across the datasets and systems studied; its table includes BLEU, ROUGE, and METEOR. The paper does not establish that correlations are especially weak for open-ended generation or tasks requiring domain-specific expertise. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Traditional methodologies predominantly rely on human annotation of texts (Callison-Burch, 2009), an approach considered overly demanding in terms of time and cost.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Partly supportedcites Callison-Burch, 2009 sourceExperimental — accuracy not yet measured
The Abstract and Introduction say manual evaluation of machine translation is generally thought to be time-consuming and expensive. But the Introduction says researchers routinely use automatic metrics rather than predominantly relying on human annotation, and the paper demonstrates that manual evaluation can be cheaper and faster than commonly thought. Its findings concern translation evaluation, not text annotation generally. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While a single powerful LLM can already tackle various missions, emerging studies suggest that multiple LLMs can further improve one another through debate and cooperation (Li et al., 2023a;Liang et al., 2023).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Not matchedcites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Liang et al., 2023 source
The Abstract describes multiple LLM agents debating while a judge uses their arguments to reach a final solution, and reports effective results on two tasks. The Generative Agents section says the framework can enhance performance even when the agents use the same backbone LLM. This supports the citing sentence’s hedged claim about improvement through multi-agent interaction. (LLM, conf=0.90)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the human evaluation processes, relying on a single perspective can introduce bias and instability in the results (Karpinska et al., 2021).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Supportedcites Karpinska et al., 2021 source
Section 4 reports that teachers disagreed because of differences in judgment and strictness, and that one teacher changed a previous rating in about 20% of the cases examined. Appendix C says hearing another teacher’s perspective often led raters to adjust their scores. Together with the high variance between workers reported in Section 6, these passages support the claim that relying on one evaluator’s perspective can produce unstable results. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT(SA) 0.474 0.421 0.527 0.482 0.599 0.549 0.576 0.558 0.544 0.503”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatEval surpasses FairEval's best results within both ChatGPT and GPT-4 settings showing the effectiveness of our proposed method.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.4 RESULTS FOR OPEN-ENDED QUESTION ANSWERS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, no significant upward trend is detected with respect to the increase in discussion turns, as is shown in Figure 3b.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“utilizing the same role description in the prompt can lead to a degradation in performance.”
- Also stated in: § 4.1 THE IMPORTANCE OF DIVERSE ROLE PROMPTS · plain
“ChatEval with the same role prompt design underperforms that with diverse role prompt design and cannot effectively enhance the performance compared with single-agent setting”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 THE IMPORTANCE OF DIVERSE ROLE PROMPTS · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“n-gram-based metrics and embedding-based metrics perform overall poorly on all the aspects evaluated illustrating that these methods can hardly reveal human preference.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT(MA) 0.441 0.396 0.500 0.454 0.664 0.607 0.602 0.583 0.552 0.510”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 6
- The quote only partly states this claim. 5
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This observation aligns with the findings in Liang et al. (2023); Du et al. (2023), highlighting a consistent phenomenon where continual discussion often leads to stagnation or even degradation of performance.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
What the sources say
- Supportedcites Liang et al., 2023 source
Section 4.3 reports that the highest COMET score occurred at the first iteration and that forcing debate to continue harmed translation results. It also notes that complex sentences sometimes benefit from additional rounds, so the finding is a tendency rather than a rule for every case. (LLM, conf=0.91)
- Not matchedcites Du et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost.”
- Also stated in: § 1 INTRODUCTION · plain
“Evaluating the quality of text generated by language models or written by humans has long been a challenging endeavor, consistently garnering substantial attention (Celikyilmaz et al., 2020).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
What the sources say
- Supportedcites Celikyilmaz et al., 2020 source
Section 2, Human-Centric Evaluation Methods (p. 5), says human evaluation of generated text is challenging and can be expensive and time-consuming. Section 2.1 (p. 6) adds that comparison-based evaluations can become prohibitively costly. These passages support the citing sentence’s broad characterization of text evaluation. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § 1 INTRODUCTION · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Findings from these researches indicate that LLM can mimic human behavior and provide evaluations that correspond with human judgments, revealing a potentially scalable and transparent alternative to costly and laborious human evaluations.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Not matchedcites Liu et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Zheng et al., 2023 source
The Introduction (p. 2) reports that GPT-4 judges matched human evaluations at over 80% agreement and describes LLM-as-a-judge as a scalable alternative to human evaluation. Section 3.2 (p. 4) says LLM judges provide explanations alongside scores, making their evaluations interpretable; this supports the claim’s reference to transparency in that sense. (LLM, conf=0.90)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Recognizing this, best practices often involve multiple human annotators collaborating in the evaluation (Van Der Lee et al., 2019).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Supportedcites Van Der Lee et al., 2019 source
Table 3 (Annotation) explicitly recommends recruiting multiple annotators—at least two, preferably more—for qualitative analysis. This supports the claim that best practices involve several human annotators in an evaluation, although the recommendation is specific to qualitative analysis. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, ChatEval improves the average Spearman and Kendall-Tau correlation by 0.096 (16.3%) and 0.057 (10.0%) respectively.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Annotator1 | - | 68.8 | 0.5”
- Also stated in: § Automatic NLG evaluation · plain
“Annotator2 | - | 76.3 | 0.62”
- Also stated in: § Automatic NLG evaluation · plain
“Annotator3 | - | 70 | 0.5”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Automatic NLG evaluation · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Multi-Agent (Same Role Prompt) | 53.8 | 0.25”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the end of the debate, we finally extract the evaluation results that both responses are of the same quality which is identical to human annotation results.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.4 QUALITATIVE ANALYSIS
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 | Single-Agent 61.3 | 0.36”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Single-Agent 53.8 | 0.27”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 | Multi-Agent | 63.8 | 0.40”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the role number increases, there is a corresponding growth in Acc. and Kap.This pattern reaches an apex with an Acc. of 62.5% at role numbers 3 and 4 before declining at role number 5.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Multi-Agent | 60.0 | 0.33”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the other two communication strategies did not perform as robustly as the one-by-one strategy, it is noteworthy that they still exceeded the performance of the naive single-agent method.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.2 THE STUDY OF COMMUNICATION STRATEGIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the multi-agent-based method improves the accuracy by 6.2% for ChatGPT and 2.5% for GPT-4;”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.4 RESULTS FOR OPEN-ENDED QUESTION ANSWERS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments on two benchmark tasks illustrate that ChatEval delivers superior accuracy and correlation in alignment with human assessment.”
- Also stated in: § 1 INTRODUCTION · plain
“We propose a multi-agent-based framework called ChatEval that aligns better with human preferences compared with single-agent-based approaches as depicted in Figure 1.”
- Also stated in: § 3.4 RESULTS FOR OPEN-ENDED QUESTION ANSWERS · plain
“ChatEval can enhance the performance of the evaluation process, achieving higher alignment with human preference compared with single-agent evaluation.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we can find that the one-by-one communication strategy is more effective than other strategies for ChatGPT setting.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.2 THE STUDY OF COMMUNICATION STRATEGIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Multi-Agent (Diverse Role Prompt) 60 | 0.33”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Simultaneous-Talk | 55 | 0.28”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Simultaneous-Talk-with-Summarizer 55 | 0.27”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, compared with the single-agent method, ChatEval amplifies the performance both for ChatGPT and GPT-4, showing the effectiveness of our method which is aligned with the results in Section 3.4.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite this decline in accuracy, Kap.continues to ascend between role numbers 3 and 4, underscoring the effectiveness of incorporating diverse roles within ChatEval.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | One-by-One | 60 | 0.33”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | One-by-One | 60 | 0.33”
- Also stated in: § Automatic NLG evaluation · plain
“ChatGPT | Simultaneous-Talk | 55 | 0.28”
- Also stated in: § Automatic NLG evaluation · plain
“ChatGPT | Simultaneous-Talk-with-Summarizer 55 | 0.27”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Automatic NLG evaluation · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“employ position calibration techniques in both single-agent and multi-agent settings.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.3 BASELINES
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatEval transcends mere textual scoring, offering a humanmimicking evaluation process for reliable assessments.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
In the paper
“The results show that their method outperforms previous traditional metrics depicting the fact that the LLM-based evaluator is effective and reliable for evaluating the dialogue response generation task.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
What the sources say
- Not matchedcites Liu et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“set the temperature to 0 to ensure reproducibility.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.1 IMPLEMENTATION DETAILS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we present evidence that ChatEval contributes to improving the evaluation performance concerning text quality, aligning more closely with human preferences.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use the same prompt template as our multi-agent debate settings in single-agent baseline except that we ignore some slot.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our qualitative analysis of the discussion process conveys insightful intuitions about how a text is evaluated by ChatEval and substantiates our approach's ability to support comprehensive evaluations akin to human judgment, thereby demonstrating the reliability and efficacy of our framework.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
“We then take the human annotation results from Wu et al. (2023) to conduct the experiments in this paper.”
“We then take the human annotation results from Mehri & Eskenazi (2020) where they carry out the annotations on 60 dialogue contexts with each response generated by 6 different systems.”
The paper relies on this without support.
Majority voting among annotators and averaging scores yield an impartial and balanced evaluation.
“This methodological approach ensures the impartiality and balance of our evaluation process.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This technique significantly enhances the capabilities of LLMs for managing complex tasks, including solving mathematical problems and performing retrieval-augmented code generation11–14.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Evaluating multi-agent collaboration in LLM-based systems requires different methodologies compared to traditional reinforcement learning-driven coordination [7, 48, 89].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [7]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
“4 for m ← n to N do 5 if m > 1 then 6 H m ← H m + h n ;”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Debater Agents.
“After configuring the agents, we then start the group debate where each agent autonomously receives responses from the others and, in turn, delivers its own responses to them.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Additionally, we include results from n-gram-based metrics, such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and embedding-based metrics such as BERTScore (Zhang et al., 2019).”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Recent advancements in the field of natural language processing have led to the emergence of billion-parameter scale LLMs, such as GPT-3 (Brown et al., 2020).”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Automatic evaluation metrics based on n-grams, such as Rouge (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Baner-jee & Lavie, 2005), have been proposed to tackle this issue (Kondrak, 2005).”
MethodInfluentialInterrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · cited 1×
“(McHugh, 2012) which gauges the agreement between results from models and human annotators while taking into account the possibility of agreement occurring by chance.”
BackgroundThe Earth Mover's Distance as a Metric for Image Retrieval
Rubner, Y., Tomasi, C., & Guibas, L. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026543900054 · Cited by 5,547
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Moreover, various training paradigms have been proposed to endow LLMs with the ability to accomplish tasks in a zero-shot manner and better adhere to human-provided instructions (Ouyang et al., 2022; Sanh et al., 2021; Wei et al., 2021).”
BackgroundBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 2×
“Additionally, we include results from n-gram-based metrics, such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and embedding-based metrics such as BERTScore (Zhang et al., 2019).”
MethodEvidence for a Collective Intelligence Factor in the Performance of Human Groups
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010) · Science · doi:10.1126/science.1193147 · Cited by 2,685
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Research works printed but not linked: 2
- Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Cited by
Sorted by citations
Items 101–107 of 107
MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1351
RoPoLL: Robust Panel of LLM Judges
Acharya, A., Pan, K. W., & Verkhovsky, B. (2026) · arXiv · doi:10.48550/arxiv.2606.30931
Song, M., Kim, C., Eo, S., & Park, C. (2026) · arXiv · doi:10.48550/arxiv.2608.30373
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Liu, Y., James, J., Wang, Y., Zhao, K., Xiao, C., & Lin, C. (2026) · arXiv · doi:10.48550/arxiv.2610.07109
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
Rao, D., & Callison-Burch, C. (2026) · arXiv · doi:10.48550/arxiv.2606.00093
Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges
Motger, Q., Oriol, M., Marco, J., & Franch, X. (2026) · arXiv · doi:10.48550/arxiv.2607.26212 · Cited by 5
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594 · Cited by 1,781
Metadata
- Authors
- Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z.
- Affiliations
- Tsinghua University (CN)
- Peking University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2308.07201
- OpenAlex
- W4385849309
- Cited by
- 725
- References
- 49
- Citations by year
- 2026: 253 · 2025: 269 · 2024: 181 · 2023: 21
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- LLM-based evaluation, multi-agent debate, text evaluation, large language models, human evaluation, multi-agent systems, natural language generation, open-ended questions, model quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
