The DOI matches its DataCite record; 63 of the 63 references it cites were found in a registry or an index; 8 of the 13 sources its claims cite were checked for support.
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Open accessAbstract
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
ChatEval is a multi-agent debate framework for LLM-based text evaluation, using multiple LLM agents with diverse role prompts and different communication strategies to discuss and judge generated responses. Experiments on FairEval and Topical-Chat show improved alignment with human assessment over single-agent evaluation and several baselines.
Key findings
- On FairEval open-ended question answering, multi-agent ChatEval improved accuracy over single-agent evaluation by 6.2% for ChatGPT (53.8% → 60.0%) and 2.5% for GPT-4 (61.3% → 63.8%), surpassing FairEval’s best MEC+BPC results (ChatGPT 58.7%, GPT-4 62.5%). Human annotators averaged 71.7% accuracy, with individual annotators at 68.8%, 76.3%, and 70.0%.
- On Topical-Chat dialogue response generation, ChatEval with GPT-4 improved average Spearman and Kendall-Tau correlation by 0.096 (16.3%) and 0.057 (10.0%), respectively, over G-EVAL-4.
- Diverse role prompts were essential: with the same role prompt, multi-agent ChatGPT scored 53.8% accuracy and 0.25 Kappa, versus 60.0% and 0.33 with diverse role prompts; the same-role condition did not effectively improve over single-agent (53.8%, 0.27).
- For ChatGPT on FairEval, one-by-one communication was strongest among tested strategies (60.0% accuracy, 0.33 Kappa), compared with simultaneous-talk (55.0%, 0.28) and simultaneous-talk-with-summarizer (55.0%, 0.27). Accuracy peaked at 62.5% with 3–4 roles before declining at 5 roles, while increasing discussion turns showed no significant upward trend.
Methods
- ChatEval treats each LLM as a debater agent with a distinct role/persona and a shared prompt template; agents autonomously exchange responses over discussion turns without human intervention. Final judgments use majority vote for comparison formats or average score for direct-score formats. Three communication strategies are tested: one-by-one, simultaneous-talk, and simultaneous-talk-with-summarizer.
- Evaluated on two benchmarks: FairEval for open-ended question answering (80 questions, human annotations from Wu et al. 2023, three annotators comparing Vicuna-13B and ChatGPT) and Topical-Chat for dialogue response generation (60 dialogue contexts, responses from 6 systems, human annotations from Mehri & Eskenazi 2020; four dimensions: naturalness, coherence, engagingness, groundedness).
- Uses OpenAI GPT-family models GPT-4 and ChatGPT (GPT-3.5-turbo) with temperature 0, in homogeneous groups. Default configuration is one-by-one communication, 2 agents, and 2 discussion turns, with position calibration in both single-agent and multi-agent settings. Metrics are Accuracy and Kappa for FairEval, and turn-level Spearman and Kendall-Tau correlations for Topical-Chat.
Limitations
- The paper states that it focuses on homogeneous groups of LLMs, with heterogeneous groups left for future research; it also notes that communication strategies other than one-by-one did not perform as robustly and require further exploration and optimization.
- The paper states that continual discussion often leads to stagnation or even degradation of performance, possibly due to ever-increasing context length.
- Evident from the text: evaluation is limited to two benchmarks and GPT-family models, and the paper does not report testing with non-GPT or heterogeneous model groups.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Own result · 6 Own interpretation · 5 Partly supported · 3 Supported · 3 Own method · 1 Sources not found · 1 No full text of the sources · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, these methods have been shown to exhibit a relatively weak correlation with human judgments, particularly in the context of tasks involving open-ended generation or requiring domain-specific expertise (Novikova et al., 2017).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Partly supportedcites Novikova et al., 2017 sourceExperimental — accuracy not yet measured
Section 7.1 reports that no metric achieved even a moderate utterance-level correlation with human ratings across the datasets and systems studied; its table includes BLEU, ROUGE, and METEOR. The paper does not establish that correlations are especially weak for open-ended generation or tasks requiring domain-specific expertise. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Traditional methodologies predominantly rely on human annotation of texts (Callison-Burch, 2009), an approach considered overly demanding in terms of time and cost.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Partly supportedcites Callison-Burch, 2009 sourceExperimental — accuracy not yet measured
The Abstract and Introduction say manual evaluation of machine translation is generally thought to be time-consuming and expensive. But the Introduction says researchers routinely use automatic metrics rather than predominantly relying on human annotation, and the paper demonstrates that manual evaluation can be cheaper and faster than commonly thought. Its findings concern translation evaluation, not text annotation generally. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While a single powerful LLM can already tackle various missions, emerging studies suggest that multiple LLMs can further improve one another through debate and cooperation (Li et al., 2023a;Liang et al., 2023).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Not matchedcites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Liang et al., 2023 source
The Abstract describes multiple LLM agents debating while a judge uses their arguments to reach a final solution, and reports effective results on two tasks. The Generative Agents section says the framework can enhance performance even when the agents use the same backbone LLM. This supports the citing sentence’s hedged claim about improvement through multi-agent interaction. (LLM, conf=0.90)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the human evaluation processes, relying on a single perspective can introduce bias and instability in the results (Karpinska et al., 2021).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Supportedcites Karpinska et al., 2021 source
Section 4 reports that teachers disagreed because of differences in judgment and strictness, and that one teacher changed a previous rating in about 20% of the cases examined. Appendix C says hearing another teacher’s perspective often led raters to adjust their scores. Together with the high variance between workers reported in Section 6, these passages support the claim that relying on one evaluator’s perspective can produce unstable results. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT(SA) 0.474 0.421 0.527 0.482 0.599 0.549 0.576 0.558 0.544 0.503”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatEval surpasses FairEval's best results within both ChatGPT and GPT-4 settings showing the effectiveness of our proposed method.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.4 RESULTS FOR OPEN-ENDED QUESTION ANSWERS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, no significant upward trend is detected with respect to the increase in discussion turns, as is shown in Figure 3b.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“utilizing the same role description in the prompt can lead to a degradation in performance.”
- Also stated in: § 4.1 THE IMPORTANCE OF DIVERSE ROLE PROMPTS · plain
“ChatEval with the same role prompt design underperforms that with diverse role prompt design and cannot effectively enhance the performance compared with single-agent setting”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 THE IMPORTANCE OF DIVERSE ROLE PROMPTS · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“n-gram-based metrics and embedding-based metrics perform overall poorly on all the aspects evaluated illustrating that these methods can hardly reveal human preference.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT(MA) 0.441 0.396 0.500 0.454 0.664 0.607 0.602 0.583 0.552 0.510”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 6
- The quote only partly states this claim. 5
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This observation aligns with the findings in Liang et al. (2023); Du et al. (2023), highlighting a consistent phenomenon where continual discussion often leads to stagnation or even degradation of performance.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
What the sources say
- Supportedcites Liang et al., 2023 source
Section 4.3 reports that the highest COMET score occurred at the first iteration and that forcing debate to continue harmed translation results. It also notes that complex sentences sometimes benefit from additional rounds, so the finding is a tendency rather than a rule for every case. (LLM, conf=0.91)
- Not matchedcites Du et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost.”
- Also stated in: § 1 INTRODUCTION · plain
“Evaluating the quality of text generated by language models or written by humans has long been a challenging endeavor, consistently garnering substantial attention (Celikyilmaz et al., 2020).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
What the sources say
- Supportedcites Celikyilmaz et al., 2020 source
Section 2, Human-Centric Evaluation Methods (p. 5), says human evaluation of generated text is challenging and can be expensive and time-consuming. Section 2.1 (p. 6) adds that comparison-based evaluations can become prohibitively costly. These passages support the citing sentence’s broad characterization of text evaluation. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § 1 INTRODUCTION · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Findings from these researches indicate that LLM can mimic human behavior and provide evaluations that correspond with human judgments, revealing a potentially scalable and transparent alternative to costly and laborious human evaluations.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Not matchedcites Liu et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Zheng et al., 2023 source
The Introduction (p. 2) reports that GPT-4 judges matched human evaluations at over 80% agreement and describes LLM-as-a-judge as a scalable alternative to human evaluation. Section 3.2 (p. 4) says LLM judges provide explanations alongside scores, making their evaluations interpretable; this supports the claim’s reference to transparency in that sense. (LLM, conf=0.90)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Recognizing this, best practices often involve multiple human annotators collaborating in the evaluation (Van Der Lee et al., 2019).”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 1 INTRODUCTION
What the sources say
- Supportedcites Van Der Lee et al., 2019 source
Table 3 (Annotation) explicitly recommends recruiting multiple annotators—at least two, preferably more—for qualitative analysis. This supports the claim that best practices involve several human annotators in an evaluation, although the recommendation is specific to qualitative analysis. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, ChatEval improves the average Spearman and Kendall-Tau correlation by 0.096 (16.3%) and 0.057 (10.0%) respectively.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Annotator1 | - | 68.8 | 0.5”
- Also stated in: § Automatic NLG evaluation · plain
“Annotator2 | - | 76.3 | 0.62”
- Also stated in: § Automatic NLG evaluation · plain
“Annotator3 | - | 70 | 0.5”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Automatic NLG evaluation · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Multi-Agent (Same Role Prompt) | 53.8 | 0.25”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the end of the debate, we finally extract the evaluation results that both responses are of the same quality which is identical to human annotation results.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.4 QUALITATIVE ANALYSIS
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 | Single-Agent 61.3 | 0.36”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Single-Agent 53.8 | 0.27”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 | Multi-Agent | 63.8 | 0.40”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the role number increases, there is a corresponding growth in Acc. and Kap.This pattern reaches an apex with an Acc. of 62.5% at role numbers 3 and 4 before declining at role number 5.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Multi-Agent | 60.0 | 0.33”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the other two communication strategies did not perform as robustly as the one-by-one strategy, it is noteworthy that they still exceeded the performance of the naive single-agent method.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.2 THE STUDY OF COMMUNICATION STRATEGIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the multi-agent-based method improves the accuracy by 6.2% for ChatGPT and 2.5% for GPT-4;”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.4 RESULTS FOR OPEN-ENDED QUESTION ANSWERS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments on two benchmark tasks illustrate that ChatEval delivers superior accuracy and correlation in alignment with human assessment.”
- Also stated in: § 1 INTRODUCTION · plain
“We propose a multi-agent-based framework called ChatEval that aligns better with human preferences compared with single-agent-based approaches as depicted in Figure 1.”
- Also stated in: § 3.4 RESULTS FOR OPEN-ENDED QUESTION ANSWERS · plain
“ChatEval can enhance the performance of the evaluation process, achieving higher alignment with human preference compared with single-agent evaluation.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we can find that the one-by-one communication strategy is more effective than other strategies for ChatGPT setting.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.2 THE STUDY OF COMMUNICATION STRATEGIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Multi-Agent (Diverse Role Prompt) 60 | 0.33”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Simultaneous-Talk | 55 | 0.28”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | Simultaneous-Talk-with-Summarizer 55 | 0.27”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, compared with the single-agent method, ChatEval amplifies the performance both for ChatGPT and GPT-4, showing the effectiveness of our method which is aligned with the results in Section 3.4.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite this decline in accuracy, Kap.continues to ascend between role numbers 3 and 4, underscoring the effectiveness of incorporating diverse roles within ChatEval.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | One-by-One | 60 | 0.33”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT | One-by-One | 60 | 0.33”
- Also stated in: § Automatic NLG evaluation · plain
“ChatGPT | Simultaneous-Talk | 55 | 0.28”
- Also stated in: § Automatic NLG evaluation · plain
“ChatGPT | Simultaneous-Talk-with-Summarizer 55 | 0.27”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Automatic NLG evaluation · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“employ position calibration techniques in both single-agent and multi-agent settings.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.3 BASELINES
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatEval transcends mere textual scoring, offering a humanmimicking evaluation process for reliable assessments.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Abstract
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
In the paper
“The results show that their method outperforms previous traditional metrics depicting the fact that the LLM-based evaluator is effective and reliable for evaluating the dialogue response generation task.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.5 RESULTS FOR DIALOGUE RESPONSE GENERATION
What the sources say
- Not matchedcites Liu et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“set the temperature to 0 to ensure reproducibility.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > 3.1 IMPLEMENTATION DETAILS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we present evidence that ChatEval contributes to improving the evaluation performance concerning text quality, aligning more closely with human preferences.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use the same prompt template as our multi-agent debate settings in single-agent baseline except that we ignore some slot.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our qualitative analysis of the discussion process conveys insightful intuitions about how a text is evaluated by ChatEval and substantiates our approach's ability to support comprehensive evaluations akin to human judgment, thereby demonstrating the reliability and efficacy of our framework.”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The human annotation results taken from prior benchmarks are a suitable reference standard for human preference.
“We then take the human annotation results from Wu et al. (2023) to conduct the experiments in this paper.”
“We then take the human annotation results from Mehri & Eskenazi (2020) where they carry out the annotations on 60 dialogue contexts with each response generated by 6 different systems.”
The paper relies on this without support.
Majority voting among annotators and averaging scores yield an impartial and balanced evaluation.
“This methodological approach ensures the impartiality and balance of our evaluation process.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This technique significantly enhances the capabilities of LLMs for managing complex tasks, including solving mathematical problems and performing retrieval-augmented code generation11–14.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Evaluating multi-agent collaboration in LLM-based systems requires different methodologies compared to traditional reinforcement learning-driven coordination [7, 48, 89].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [7]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Automatic NLG evaluation
“4 for m ← n to N do 5 if m > 1 then 6 H m ← H m + h n ;”
§ CHATEVAL: TOWARDS BETTER LLM-BASED EVALUA-TORS THROUGH MULTI-AGENT DEBATE > Debater Agents.
“After configuring the agents, we then start the group debate where each agent autonomously receives responses from the others and, in turn, delivers its own responses to them.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Additionally, we include results from n-gram-based metrics, such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and embedding-based metrics such as BERTScore (Zhang et al., 2019).”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Recent advancements in the field of natural language processing have led to the emergence of billion-parameter scale LLMs, such as GPT-3 (Brown et al., 2020).”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Automatic evaluation metrics based on n-grams, such as Rouge (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Baner-jee & Lavie, 2005), have been proposed to tackle this issue (Kondrak, 2005).”
MethodInfluentialInterrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · cited 1×
“(McHugh, 2012) which gauges the agreement between results from models and human annotators while taking into account the possibility of agreement occurring by chance.”
BackgroundThe Earth Mover's Distance as a Metric for Image Retrieval
Rubner, Y., Tomasi, C., & Guibas, L. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026543900054 · Cited by 5,547
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Moreover, various training paradigms have been proposed to endow LLMs with the ability to accomplish tasks in a zero-shot manner and better adhere to human-provided instructions (Ouyang et al., 2022; Sanh et al., 2021; Wei et al., 2021).”
BackgroundBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 2×
“Additionally, we include results from n-gram-based metrics, such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and embedding-based metrics such as BERTScore (Zhang et al., 2019).”
MethodEvidence for a Collective Intelligence Factor in the Performance of Human Groups
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010) · Science · doi:10.1126/science.1193147 · Cited by 2,685
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Research works printed but not linked: 2
- Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Cited by
Sorted by citations
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Self-citationThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“An example is cooperation, which is achieved through volunteer behaviors [330], brainstorming discussions [327, 340], and project management [332].”
InfluentialSelf-citationChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.810 · Cited by 490
Self-citationGudiño, J., Grandi, U., & Hidalgo, C. A. (2019) · doi:10.1098/rsta · Cited by 393
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
“On-device LLMs boost enterprise privacy and cost-efficiency, improving tasks like message completion, meeting summaries, and computer operations.”
Enhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 147
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 1,202
“In contrast, our work and work in (Liang et al., 2023; Chan et al., 2023) focus on using debate at inference time to improve the performance of language models.”
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.969 · Cited by 87
Self-citationWhen Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72
“, 2023), debate (Chan et al., 2023; Liang et al., 2023b; Du et al., 2023), improved safety (Irving et al.”
BackgroundLarge Language Model based Multi-Agents: A Survey of Progress and Challenges
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X. (2024) · arXiv · doi:10.48550/arxiv.2402.01680 · Cited by 1,713
On scalable oversight with weak LLMs judging strong LLMs
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04622 · Cited by 66
Stance Detection with Collaborative Role-Infused LLM-Based Agents
Lan, X., Gao, C., Jin, D., & Li, Y. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31360 · Cited by 69
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Chen, J., Saha, S., & Bansal, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.381 · Cited by 64
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
Assessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57
“…from traditional human annotation to reference-based automated methods, and now, to meth-ods based on LLMs. LLMs are increasingly playing the role of judges in tasks such as question-answering, translation, and text quality assessment[35, 36], giving rise to various evaluation framework[37].”
BackgroundThe Virtual Lab: AI Agents Design New SARS-CoV-2 Nanobodies with Experimental Validation
Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E., & Zou, J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.11.623004 · Cited by 56
Liu, Y., Lo, S. K., Lu, Q., Zhu, L., Zhao, D., Xu, X., Harrer, S., & Whittle, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112278 · Cited by 51
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40
Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025) · Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2025.ijcnlp-long.18 · Cited by 39
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Can LLMs Replace Manual Annotation of Software Engineering Artifacts?
Ahmed, T., Dévanbu, P., Treude, C., & Pradel, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00086 · Cited by 105
“LLMs have also been proposed as judges of outputs generated by other LLMs, either with a single LLM as the judge [67] or with a set of LLMs that discuss until reaching an agreement [68].”
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12708 · Cited by 34
“Our study aligns with recent works in agent-based modeling (ABM) [12, 58–61] that leverage the capabilities of LLM agents to simulate realistic environments for scientific research.”
ResultAgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 33
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
Yang, X., Li, T., Wang, H., Zhang, R., Ni, Z., Liu, N., Zhai, H., Zhao, J., Meng, F., Zhou, Z., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01486-5 · Cited by 30
“The diagnostic potential of LLMs with post-training technology warrants further exploration 41 – 45 .”
Self-citationRetrieval-based Evaluation for LLMs: A Case Study in Korean Legal QA
Ryu, C., Lee, S., Pang, S., Choi, C., Choi, H., Min, M., & Sohn, J.-Y. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.13 · Cited by 32
“Recall that existing LLM-based evaluators (Chan et al., 2023; Liu et al., 2023; Wang et al., 2023b) ask LLMs for the evaluation, where the prompts include Q and A , along with the evaluation criteria and methods for evaluation.”
MethodInfluentialMitigating reasoning hallucination through Multi-agent Collaborative Filtering
Shi, J., Zhao, J., Wu, X., Xu, R., Jiang, Y.-H., & He, L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125723 · Cited by 34
Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives
Zhang, W., Shen, Y., Wu, L., Peng, Q., Wang, J., Zhuang, Y., & Lu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.197 · Cited by 27
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
“[ 74] have conceptualized assemblies of agents as a group and focused on exploring the potential of their cooperation [18, 41, 105, 132] found social behaviors autonomously emerge within a group of agents.”
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 386
“Other similar works include (Chan et al., 2023).”
Wu, M., Xu, J., Yuan, Y., Haffari, G., Wan, L., Luo, W., & Zhang, K. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.25 · Cited by 91
Combining Multiple Large Language Models Improves Diagnostic Accuracy
Barabucci, G., Shia, V., Chu, E., Harack, B., Laskowski, K., & Fu, N. (2024) · NEJM AI · doi:10.1056/aics2400502 · Cited by 24
HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation toolPreprint
Awasthi, R., Mishra, S. V., Mahapatra, D., Khanna, A. K., Maheshwari, K. K., Cywiński, J. B., Papay, F., & Mathur, P. (2023) · medRxiv · doi:10.1101/2023.12.22.23300458 · Cited by 23
“Assessing LLMs is a challenging task, as there is no one-size-fits-all evaluation method (Zhang et al. [2023], Chan et al. [2023]).”
MethodImproving Multi-Agent Debate with Sparse Communication Topology
Li, Y., Du, Y., Zhang, J., Hou, L., Grabowski, P., Li, Y., & Ie, E. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.427 · Cited by 20
Large language models (LLMs) as agents for augmented democracy
Gudiño, J., Grandi, U., & Hidalgo, C. A. (2024) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0100 · Cited by 27
Automating quantum computing laboratory experiments with an agent-based AI framework
Cao, S., Zhang, Z., Alghadeer, M., Fasciati, S. D., Piscitelli, M., Bakr, M. S., Leek, P., & Aspuru‐Guzik, A. (2025) · Patterns · doi:10.1016/j.patter.2025.101372 · Cited by 55
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
“Agents can re-engage or retry if inconsistencies arise, as shown in Appendix I.5 [51].”
Diagnosing Bias and Instability in LLM Evaluation: A Scalable Pairwise Meta-Evaluator
Anghel, C., Anghel, A., Pecheanu, E., Cocu, A., Istrate, A., & Andrei, C. (2025) · Inf. · doi:10.3390/info16080652 · Cited by 20
Amayuelas, A., Yang, X., Antoniades, A., Hua, W., Pan, L., & Wang, W. Y. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.407 · Cited by 15
Lee, P., Son, M., & Jia, Z. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10067-y · Cited by 19
“This approach is particularly effective for reaching consensus or refining solutions, as demonstrated in scientific debate simulations (e.g., Du et al., 2023; Chan et al., 2023).”
War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars
Hua, W., Fan, L., Li, L., Mei, K., Ji, J., Ge, Y., Hemphill, L., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2311.17227 · Cited by 15
Project Sid: Many-agent simulations toward AI civilization
AL, A., Ahn, A., Becker, N., Carroll, S., Christie, N., Cortes, M., Demirci, A., Du, M., Li, F., Luo, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.00114 · Cited by 12
“Multi-agent frameworks have been deployed in software development (43; 27), cooperative robotic control (60), scientific experiments (12; 47), and debates (3).”
BackgroundGenerative AI Agents for Knowledge Work Augmentation in Finance
Ganesh, S., Ardon, L., Borrajo, D., Garg, D., Madhushani, U., Narayanan, A. L., Canonaco, G., & Veloso, M. (2024) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-022823-033813 · Cited by 13
The future of cognitive strategy-enhanced persuasive dialogue agents: new perspectives and trends
Chen, M., Guo, B., Wang, H., Li, H., Zhao, Q., Liu, J., Ding, Y., Pan, Y., & Yu, Z. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40057-x · Cited by 11
ChatLLM network: More brains, more intelligence
Hao, R., Hu, L., Qi, W., Wu, Q., Zhang, Y., & Nie, L. (2025) · AI Open · doi:10.1016/j.aiopen.2025.01.001 · Cited by 53
Plurals: A System for Guiding LLMs via Simulated Social Ensembles
Ashkinaze, J., Fry, E., Edara, N., Gilbert, É., & Budak, C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713675 · Cited by 24
“Our work builds on research in deliberation [14, 15, 32, 42, 68, 73, 102], pluralistic sociotechnical systems [3, 39, 63, 118], and multi-agent AI alignment approaches [17, 47, 58, 106, 110].”
Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
Schroeder, K., & Wood-Doughty, Z. (2024) · arXiv · doi:10.48550/arxiv.2412.12509 · Cited by 8
Characterised LLMs Affect its Evaluation of Summary and Translation
Lu, Y., & Lin, Y.-T. (2023) · doi:10.18653/v1/2023.eval4nlp-1.15 · Cited by 8
“…scores and evaluate text(Tom Kocmi, 2023)(Jinlan Fu, 2023)(Fu et al., 2023), as well as research exploring having LLMs play the role of experts(Chan et al., 2023), we present an evaluation system employing multiple metrics (Jinlan Fu, 2023) by carefully designed prompt (Tom Kocmi, 2023) and…”
Scaling Large Language Model-based Multi-Agent Collaboration
Qian, C., Xie, Z., Wang, Y., Liu, W., Zhu, K., Xia, H., Dang, Y., Du, Z., Chen, W., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07155 · Cited by 8
“( Chen et al., 2024b; Chan et al., 2024).”
InfluentialSelf-citationDebate Helps Supervise Unreliable Experts
Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.08702 · Cited by 8
Han, Z., Wang, J., Yan, X., Jiang, Z., Zhang, Y., Liu, S., Gong, Q., & Song, C. (2025) · Applied Sciences · doi:10.3390/app15105663 · Cited by 7
“ChatEval [61] employs multiple agents and debates using different communication strategies to promote the automatic evaluation of LLMs. LangGround [62] is a Multi-Agent Reinforcement Learning (MARL) computational framework that enables agents to learn human-interpretable communication languages in…”
LLM-as-a-Judge Approaches as Proxies for Mathematical Coherence in Narrative Extraction
Keith, B. (2025) · Electronics · doi:10.3390/electronics14132735 · Cited by 10
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.911 · Cited by 9
Reversible source-aware natural language watermarking via customized lexical substitution
Jiang, Z., Wang, H., Shi, Z., et al. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103977 · Cited by 5
Performance Evaluation and Comparative Ranking of LLM Variants in Entity Relationship Prediction
Vardhni, K., Devaraja, G., Dharhshita, R., et al. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10725334 · Cited by 5
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A. (2023) · arXiv · doi:10.48550/arxiv.2311.17371 · Cited by 73
Agent-as-a-Judge: Evaluate Agents with Agents
Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10934 · Cited by 5
Self-citationMNC: A multi-agent framework for complex network configuration
Wang, C., & Li, H. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200531 · Cited by 5
An LLM-driven Transcription Task for Mobile Text Entry Studies
Komninos, A., Feit, A. M., Leiva, L. A., Lehmann, F., Simou, I., Minas, D., Fotopoulos, A., & Xenos, M. (2024) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3701571.3701586 · Cited by 4
“LLMs-as-a-judge can be deployed in a single-agent, or multi-agent manner (e.g. through agentic negotiation) [8, 11, 23].”
Evaluating LLMs on Kazakhstan's mathematics exam for university admission
Kadyrov, S., Abdrasilov, B. S., Sabyrov, A., Baizhanov, N., Makhmutova, A., & Kyllonen, P. C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1642570 · Cited by 3
ScamFerret: Detecting Scam Websites Autonomously with Large Language Models
Nakano, H., Koide, T., & Chiba, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97620-9_1 · Cited by 8
DeltaScore: Fine-Grained Story Evaluation with Perturbations
Xie, Z., Li, M., Cohn, T., & Lau, J. H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.353 · Cited by 14
Optimizing the role of human evaluation in LLM-based spoken document summarization systems
Kroll, M., & Kraus, K. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2268 · Cited by 3
“ChatEval utilizes several LLMs trained as ‘agents’ to evaluate a task, engaging them in debate that the authors argue leads to better performance than a single LLM evaluation [9].”
MethodFree-MAD: Consensus-Free Multi-Agent Debate
Cui, Y., Fu, H., Zhang, H., Wang, L., & Zuo, C. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1600 · Cited by 28
Imitating Mistakes in a Learning Companion AI Agent for Online Peer Learning
Moribe, S., & Ushiama, T. (2025) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom64595.2025.10857528 · Cited by 2
“ChatEval[19] shows that a multi-agent system can accurately evaluate sentences by enabling multiple agents, each assigned a distinct role in evaluating input text, to interact and debate.”
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 24
Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
Zhu, X., Zhang, C., Chi, Y., Stafford, T., Collier, N., & Vlachos, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1694 · Cited by 28
Designing a Distributed LLM-Based Search Engine as a Foundation for Agent Discovery
Nilofary, P., Feghhi, M., & Analoui, M. (2025) · International Computer Conference, Computer Society of Iran (CSICC) · doi:10.1109/csicc65765.2025.10967406 · Cited by 1
“This concept is exemplified by the multi-agent debate (MAD) framework, which employs multiple LLM agents to discuss and debate with each other, generating and updating responses based on their solutions [14] [15].”
Tracking Evolving Relationship Between Characters in Books in the Era of Large Language Models
Sancheti, A., & Rudinger, R. (2025) · Workshop on Narrative Understanding (WNU) · doi:10.18653/v1/2025.wnu-1.12 · Cited by 3
AlignLLM: Alignment-Based Evaluation Using Ensemble of LLMs-as-Judges for Q&A
Abeyratne, R., Wiratunga, N., Martin, K., Nkisi-Orji, I., & Jayawardena, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96559-3_2 · Cited by 2
Kraidia, I., Qaddara, I., Almutairi, A., Alzaben, N., & Belhouari, S. B. (2026) · Scientific Reports · doi:10.1038/s41598-026-42705-7 · Cited by 8
“Debate-based approaches, in particular, allow agents to iteratively critique and refine one another’s arguments, leading to improvements in factuality and logical consistency on complex reasoning tasks 2,14 .”
Fang, W., Li, L., Shi, K., Yang, Y., & Zhang, J. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792976 · Cited by 1
ConfidenceCal: Enhancing LLMs Reliability through Confidence Calibration in Multi-Agent Debate
Bai, Y. (2024) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia63733.2024.10808396 · Cited by 2
Improving Spam Detection with a Multi-Agent Debate Framework
Huang, R. (2024) · IEEE International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA) · doi:10.1109/iciba62489.2024.10868417 · Cited by 1
DAFE: LLM-Based Evaluation Through Dynamic Arbitration for Free-Form Question-AnsweringPreprint
Badshah, S., & Sajjad, H. (2025) · Qeios · doi:10.32388/b69sky · Cited by 1
Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate
Wynn, A., Satija, H., & Hadfield, G. (2025) · arXiv · doi:10.48550/arxiv.2509.05396 · Cited by 96
“…line of research on interactive reasoning among multiple LLMs through debate has promoted the multi-agent debate framework as a promising approach to enhancing the reasoning and decision-making capabilities of LLM agents (Du et al., 2023; Chan et al., 2023; Liang et al., 2023; Khan et al., 2024).”
Bayesian Calibration of Win Rate Estimation with LLM Evaluators
Gao, Y., Xu, G., Wang, Z., & Cohan, A. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.273 · Cited by 1
“…applied various prompting techniques to improve the accuracy of LLM evaluation, including chain of thought (Liu et al., 2023a), evaluation with explanation (Chiang and Lee, 2023b), multi-LLM discussion (Chan et al., 2023; Li et al., 2023), and calibration with human expert (Liu et al., 2023b).”
MethodFormalAlign: Automated Alignment Evaluation for Autoformalization
Lu, J., Wan, Y., Huang, Y., Xiong, J., Liu, Z., & Guo, Z. (2024) · arXiv · doi:10.48550/arxiv.2410.10135 · Cited by 1
Self-citationCan LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
Wu, H., Li, Z., & Li, L. (2025) · arXiv · doi:10.48550/arxiv.2511.07784 · Cited by 1
“Recent studies show that large language models (LLMs) can achieve similar benefits through debate (Estornell and Liu, 2024; Liang et al., 2024; Chan et al., 2023; Liu et al., 2025b).”
InfluentialDo as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1
Gómez Álvarez, S., Mozo Quesada, A., Navarro, T., Gálvez Rojas, S., & López Valverde, F. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01086-z · Cited by 1
HIT-MI&T Lab’s Submission to Eval4NLP 2023 Shared Task
Zhang, R., Song, F., Huang, H., Yuan, J., Yang, M., & Zhao, T. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.11
OSAgent: Copiloting Operating System with LLM-based Agent
Xu, J., Guo, K., Gong, W., & Shi, R. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650304 · Cited by 1
“Another line of work involves the development of LLM-based Agents [9]–[13], including HuggingGPT [14] and AutoGPT [15], which extend LLM capabilities through external memory and the utilization of auxiliary tools.”
BackgroundMulti-agent collaborative crowd intelligence persuasive dialogue inspired by cognitive psychology
Mengqi, C., Guo, B., Wang, H., Jingqi, L. I. U., Zhao, K., Liang, Y., Yan, P. A. N., & Wu, Z. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0204
Large Language Models Are Human-Like Annotators
Marreddy, M., Oota, S. R., & Gupta, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_45
Enhancing Telecom Operation Support Systems with Multi-Agent Large Language Models
Lee, K.-Y., Tseng, P.-A., & Chen, W.-C. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00055
“Multi-agent[8][9] systems offer enhanced scalability by distributing tasks across various agents, allowing for better handling of large-scale operations and complex tasks.”
Self-citationKnowledge Intensive AgentsPreprint
Liu, Z., Liu, Z., Huang, P., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5459034
Self-citationLLM-as-a-Judge for mental support: A meta-evaluation using domain-specific platform data
Li, Y., Chen, X., & Dolata, M. (2026) · Electronic Markets · doi:10.1007/s12525-026-00890-3
A modular tool-driven RAG framework for engineering test documentation
Trappey, A. J. C., Huang, W.-Y., & Chien, C.-H. (2026) · Journal of Engineering Design · doi:10.1080/09544828.2026.2698389
Efficient LLM Safety Evaluation through Multi-Agent Debate
Lin, D., Shen, G., Yang, Z., Liu, T., Zhao, D., & Zeng, Y. (2025) · arXiv · doi:10.48550/arxiv.2511.06396 · Cited by 8
SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
Li, D., Tan, Z., Qian, P., Li, Y., Chaudhary, K. S., Hu, L., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.03284 · Cited by 35
Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Ferreira, L., Liu, G., & Zheng, K. (2026) · arXiv · doi:10.48550/arxiv.2609.35875
“Extensions include evaluation (Chan et al., 2023) and divergent-thinking prompting (Liang et al., 2023).”
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Li, X., Liu, M., & Yuen, C. (2026) · arXiv · doi:10.48550/arxiv.2609.35279
At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent
Dylan, D., Brennan, A., Murphy, C., O'Sullivan, N., Kelly, C., & Walsh, S. (2026) · arXiv · doi:10.48550/arxiv.2609.04217 · Cited by 3
Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity
Zhang, H., Cui, Z., Chen, J., Wang, X., Zhang, Q., Wang, Z., Wu, D., & Hu, S. (2025) · arXiv · doi:10.48550/arxiv.2502.08788 · Cited by 84
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Liu, M. T.-Y., Harrington, A., Darrell, T., & Bai, Y. (2026) · arXiv · doi:10.48550/arxiv.2610.21179
Metadata
- Authors
- Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z.
- Affiliations
- Tsinghua University (CN)
- Peking University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2308.07201
- OpenAlex
- W4385849309
- Cited by
- 725
- References
- 49
- Citations by year
- 2026: 253 · 2025: 269 · 2024: 181 · 2023: 21
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- LLM-based evaluation, multi-agent debate, text evaluation, large language models, human evaluation, multi-agent systems, natural language generation, open-ended questions, model quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
