The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index; 6 of the 14 sources its claims cite were checked for support.
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Open accessAbstract
Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks, which drives the research on cognitive behaviors of LLMs to explore human-like problem-solving strategies.Along this direction, one representative strategy is self-reflection, which asks an LLM to refine the solution with the feedback generated by itself iteratively.However, our study shows that such reflection-style methods suffer from the Degeneration-of-Thought (DoT) problem: once the LLM has established confidence in its solutions, it is unable to generate novel thoughts later through reflection even if its initial stance is incorrect.To address the DoT problem, we propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution.Clearly, our MAD framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation.Experiment results on two challenging datasets, commonsense machine translation and counterintuitive arithmetic reasoning, demonstrate the effectiveness of our MAD framework.Extensive analyses suggest that the adaptive break of debate and the modest level of "tit for tat" state are required for MAD to obtain good performance.Moreover, we find that LLMs might not be a fair judge if different LLMs are used for agents.Code is available at https://github. com/Skytliang/Multi-Agents-Debate.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper identifies a Degeneration-of-Thought (DoT) problem in LLM self-reflection, where models stick to incorrect initial answers, and proposes Multi-Agent Debate (MAD) with debaters and a judge to encourage divergent thinking. MAD improves over self-reflection and other baselines on commonsense machine translation and counter-intuitive arithmetic reasoning, though it adds inference cost and depends on debate design choices.
Key findings
- On Common MT, GPT-3.5-Turbo with MAD surpasses GPT-4 on automatic and human metrics: lexical COMET 82.0 vs. 82.0, BLEURT 70.9 vs. 70.1, HUMAN 3.78 vs. 3.41; contextless COMET 84.8 vs. 84.7, BLEURT 73.7 vs. 73.6, HUMAN 3.67 vs. 3.63; contextual COMET 85.3 vs. 85.0, BLEURT 74.0 vs. 73.7, HUMAN 3.67 vs. 3.65.
- On Counter-Intuitive AR, MAD with GPT-3.5-Turbo reaches 37.0% accuracy, outperforming GPT-3.5-Turbo (26.0), CoT (28.0), Self-Consistency (29.5), and Self-Reflect (27.5), but not GPT-4 (51.0).
- MAD mitigates DoT relative to Self-Reflect: bias decreases from 29.0 to 24.8, and diversity increases from 19.3 to 49.7.
- Analyses show adaptive break and a modest “tit for tat” level are needed for good performance; increasing debaters from 2 to 3 to 4 reduces Common MT performance (COMET 84.4 → 83.1 → 82.9; HUMAN 3.69 → 3.58 → 3.49), and LLM judges may favor the side sharing their backbone LLM.
Methods
- MAD uses multiple debaters who argue in a fixed order under “tit for tat” meta prompts, plus a judge with discriminative mode to stop adaptively or extractive mode to produce a final answer; experiments set a maximum of 3 debate iterations.
- Tasks: Commonsense Machine Translation (Chinese→English; 200 lexical, 450 contextless syntactic, 350 contextual syntactic examples) and Counter-Intuitive Arithmetic Reasoning (200 questions). Backbones include vicuna-7b-v1.5-16k, vicuna-13b-v1.5-16k, GPT-3.5-Turbo-0301, and GPT-4-0314; zero-shot with temperature 0.
- Metrics: COMET, BLEURT, and human 1–5 scores for Common MT; accuracy for Counter-Intuitive AR. Baselines include Self-Reflect, Rerank, MAPS, CoT, and Self-Consistency.
Limitations
- Stated: MAD requires more time cost because agents engage in multiple rounds; current LLM-based agents may struggle with coherence and relevance in long contexts; LLM-based judges may prefer outputs generated by themselves, so the paper recommends using the same LLM for all roles or distinct LLMs for judge and debaters.
- Evident from the text: scaling to more debaters reduced performance, which the authors attribute to LLMs’ limitations in handling long texts and judges’ difficulty extracting information from longer debates.
- Evident from the text: the main results are reported without variance, confidence intervals, or significance tests.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 Own result · 11 No basis given · 9 Own interpretation · 4 Partly supported · 4 Sources not found · 3 Own method · 2 Supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“we find that LLMs might not be a fair judge if different LLMs are used for agents.”
- Also stated in: § 4.2 Analysis of Judge · hedged
“This bias indicates that LLMs might not be a fair judge (Wang et al., 2023) when different LLMs are used for the agents.”
- Also stated in: § 6 Conclusion · hedged
“More interestingly, we find that LLMs might not be a fair judge if different LLMs are used for agents.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Abstract
What the sources say
- Supportedcites Wang et al., 2023 source
Section 2.2 reports that GPT-4 and ChatGPT gave different evaluation results when responses from two different assistant models changed positions. Section 3 says this positional bias can make LLMs unfair evaluators, supporting the citing sentence's qualified claim. (LLM, conf=0.90)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.2 Analysis of Judge · hedged
Every source the paper cites for this claim was checked and supports it.
In the paper
“This suggests that the key challenge of MAD with more debaters lies in the limitations of the LLMs to handle long texts (Liu et al., 2024).”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.3 Analysis of Debaters
What the sources say
- Supportedcites Liu et al., 2024 source
The Conclusion reports that language models struggle to access and use information in long input contexts, supporting the cited premise about LLM limitations. The claim that this is the key challenge for MAD with more debaters is the citing authors’ inference, not a finding attributed to this source. (LLM, conf=0.88)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can find that the baseline GPT-3.5-Turbo(even the more powerful GPT-4) incorrectly translates the source words literally.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.3 Results on Common MT
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When Turbo debaters are involved, Vicuna, serving as the judge, underperforms Turbo across all test sets.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.2 Analysis of Judge
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Strong debaters with a weak judge work better than the reverse.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.2 Analysis of Judge
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results demonstrate that our MAD framework outperforms the baseline methods”
- Also stated in: § Abstract · boosted
“Experiment results on two challenging datasets, commonsense machine translation and counterintuitive arithmetic reasoning, demonstrate the effectiveness of our MAD framework.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 1 Introduction
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when translating more complex sentences (subsets with lower human scores), the judge requires additional iterations to gather adequate information from the debaters before making a final decision.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.3 Analysis of Debaters
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3.5-Turbo with MAD can surpass the performance of GPT-4 on Common MT.”
- Also stated in: § 1 Introduction · plain
“find that GPT-3.5-Turbo with MAD can even surpass GPT-4 on the Common MT dataset.”
- Also stated in: § 6 Conclusion · boosted
“We demonstrate the effectiveness of MAD on two challenging tasks and find that GPT-3.5-Turbo with MAD can even surpass GPT-4 on the Common MT dataset.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 1 Introduction
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The quality of the debaters' responses significantly impact the performance ceiling of MAD.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.2 Analysis of Judge
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the judge shows a preference to the side with the same LLM as the backbone.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.2 Analysis of Judge
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 20
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Abstract
What the sources say
- Not matchedcites Zhu et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Gou et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction describes remarkable progress across language tasks, and §4 evaluates models on mathematical and multi-hop reasoning. But the paper’s explicit finding about intricate reasoning concerns models’ difficulty verifying and correcting their own answers (Appendix D.1), not a general inability to perform complex reasoning tasks. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“There are various factors (Bortolotti, 2011;Keestra, 2017) that could result in DoT, and we outline three here: (1) Bias and Distorted Perception.”
- Also stated in: § 1 Introduction · hedged
“(2) Rigidity and Resistance to Change.Self-reflection often involves challenging one's beliefs, assumptions, and behaviors.”
- Also stated in: § 1 Introduction · hedged
“(3) Limited External Feedback.Selfreflection is primarily an internal process, but external feedback can provide valuable perspectives and insights.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 1 Introduction
What the sources say
- Partly supportedcites Bortolotti, 2011 sourceExperimental — accuracy not yet measured
In “Decision-making and self-knowledge” (page not provided), Bortolotti says reflective deliberation may be manipulated by biased evidence and may overlook factors relevant to a choice, supporting the bias and distorted-perception factor. The passages do not identify rigidity, resistance to change, or limited external feedback as factors. (LLM, conf=0.83)
- Not matchedcites Keestra, 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · hedged
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.1 Challenging Testbeds
What the sources say
- Not matchedcites Roy and Roth, 2015
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Cobbe et al., 2021 source
The citation identifies GSM8K as a comparison dataset; it does not attribute the Counter-Intuitive AR dataset’s two challenges to Cobbe et al. The Introduction identifies GSM8K as a grade-school math word-problem dataset, supporting that use of the citation. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Concurrent with our work, a few studies (Xiong et al., 2023;Du et al., 2023) also explore the multi-agent debate framework to enhance the reasoning ability of LLMs.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Generative Agents.
What the sources say
- Supportedcites Xiong et al., 2023 source
Section 3 describes FORD as a debate framework in which multiple LLMs collaborate on shared tasks. The Conclusion reports that debates can improve LLM performance; the paper uses commonsense reasoning tasks to test this, supporting its citation as an example of the approach. (LLM, conf=0.90)
- Not matchedcites Du et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, addressing these biases and stereotypes through self-reflection can be challenging.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.1 Mitigation of DoT
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our MAD framework, though not as good as GPT-4, outperforms all the other compared methods based on GPT-3.5-Turbo”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.4 Results on Counter-Intuitive AR
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-Reflect only marginally improves over the baseline GPT-3.5-Turbo, while CoT and Self-Consistency bring more improvements.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.4 Results on Counter-Intuitive AR
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The low disagreement of self-reflection suggests that the LLM sticks to the incorrect answers predicted by CoT and is unable to engage in meaningful selfreflection.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 1 Introduction
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MAPS and Self-Reflec achieve improvements over baseline GPT-3.5-Turbo.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.3 Results on Common MT
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Complex questions require more iteration rounds of debate.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.3 Analysis of Debaters
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MAD framework effectively corrects inherent biases in translation, mitigates DoT, and considerably improves performance.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.1 Mitigation of DoT
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The questions in our dataset are designed to stimulate counter-intuitive thinking, which aims to challenge conventional decision-making by presenting situations where the immediate, intuitive response is often incorrect.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Components Content Question
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate the effectiveness of MAD on two challenging tasks and find that GPT-3.5-Turbo with MAD can even surpass GPT-4 on the Common MT dataset.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 6 Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Complex samples require more rounds of debate.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 6 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LLM-based judge may have a preference for outputs generated by itself.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A limitation of this work is that our method requires more time cost, as agents need to engage in multiple rounds of interaction to present and refute arguments.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the highest COMET score appears at the first iteration and is also lower than the result of the adaptive break.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.3 Analysis of Debaters
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Each correct answer within the dataset requires a rigorous multi-step reasoning process, thereby evaluating the capacity of LLMs to engage in complex decision-making and problemsolving.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > Components Content Question
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, LLMs made a common mistake initially in understanding the concept of average speed.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > F.2 Counter-Intuitive Arithmetic Reasoning
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The questions in our dataset are embedded in hidden traps designed to elicit intuitive and appealing answers that are often incorrect.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > A.2 Counter-Intuitive Arithmetic Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, MAD is less susceptible to the factors of DoT, and can explore divergent chain-of-thoughts to achieve accurate answers.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Within the challenging part of Common MT, the authentic translation of each source sentence requires a proper understanding of common sense knowledge.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.1 Challenging Testbeds
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The Common MT dataset is composed of Chinese⇒English translation examples (He et al., 2020), which are used to examine three types of ambiguity resolution abilities of translation models, covering lexical and contextless/contextual syntactic ambiguity.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 3.1 Challenging Testbeds
What the sources say
- Not matchedcites He et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“While self-reflection can be effective in creating better answers, it is highly dependent on the self-evaluation capabilities of LLMs, which are not formally guaranteed (Shinn et al., 2024).”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 1 Introduction
What the sources say
- Not matchedcites Shinn et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We randomly shuffled the display order of the translated sentences from all methods in an anonymous manner.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > B Human Evaluation Details
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Within the challenging part of Common MT, the authentic translation of each source sentence requires a proper understanding of common sense knowledge.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 他喜欢吃苹果。
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, we asked the annotators to label each sentence as 1 (not conforming to commonsense) or 0 (conforming to commonsense), and report the degree of bias for the whole test set.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > B Human Evaluation Details
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In response to this issue, our MAD method helps LLMs understand semantics correctly and generate highly authentic translations.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > F.1 Commonsense Machine Translation
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Then, employed three professional human translators (Krippendorff's Alpha = 0.76) to directly assess all methods together.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > B Human Evaluation Details
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Failure to identify and address such ambiguities may result in inaccurate translations.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 他喜欢吃苹果。
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our MAD method encourages divergent thinking and effectively prevents intuitive but incorrect responses, leading to the correct method for calculating average speed.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > F.2 Counter-Intuitive Arithmetic Reasoning
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The Common MT dataset is composed of Chinese⇒English translation examples (He et al., 2020), which are used to examine three types of ambiguity resolution abilities of translation models.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 他喜欢吃苹果。
What the sources say
- Not matchedcites He et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Previous studies on thinking hierarchy (Daniel, 2017) suggest that we humans have a fast and intuitive system and a slow and logical system, and tend to run the lower level system before the higher level one.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > A.2 Counter-Intuitive Arithmetic Reasoning
What the sources say
- Not matchedcites Daniel, 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Forcing the debate to continue will harm the translation results, which demonstrates the reasonableness of our adaptive break strategy.”
§ Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate > 4.3 Analysis of Debaters
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, MP was evaluated against CoT solely on the CIAR dataset (Liang et al., 2024), which was initially proposed and has not been fully disclosed.”
From Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity (2025) · cites this paper as Liang et al., 2024
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“While the idea of independent answer generation has been previously explored in different multi-agent settings (Du et al., 2023b;Liang et al., 2024), these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.”
From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Liang et al., 2024
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Consequently, each round of dialogue in the collaborative process refines artifacts from different aspects, naturally elevating the probability of producing more nuanced artifacts (Liang et al., 2024; Du et al., 2024a; Cohen et al., 2023).”
From Scaling Large Language Model-based Multi-Agent Collaboration (2024) · cites this paper as Liang et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Along this line, multi-agent collaboration has proven beneficial in uniting the expertise of diverse agents for autonomous task-solving”
From Scaling Large Language Model-based Multi-Agent Collaboration (2024) · cites this paper as Liang et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In parallel, Multi-Agent Debate (MAD) has been proposed as a way to improve the reasoning and factuality of LLM outputs by allowing multiple agents to exchange and revise their answers (Du et al., 2024; Liang et al., 2024).”
From Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation (2026) · cites this paper as Liang et al., 2024
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such applications are supported by evidence that MAD can be effective in reasoning-oriented tasks where answers are relatively verifiable, such as factual question answering or mathematical reasoning (Liang et al., 2024; Zhang and Xiong, 2025).”
From Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation (2026) · cites this paper as Liang et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Agents in debate can elicit more truthful predictions than single-agent prediction (Du et al., 2024;Liang et al., 2024)”
From Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies (2025) · cites this paper as Liang et al., 2024
In the citing paper
“This framework explains conflicting findings in prior work (Du et al., 2024; Liang et al., 2024; Chen et al., 2024; Smit et al., 2024; Wang et al., 2024b; Kaesberg et al., 2025) as arising from tasks operating in different regimes.”
From Multi-agent discussion gains less when dissent is withheld (2026) · cites this paper as Liang et al., 2024
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Self-citationSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Self-citationGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 1,202
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
Self-citationSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702
Research works printed but not linked: 3
- Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few- shot learning for machine translation. In Inter- national Conference on Machine Learning, pages 10867-10878.
- PMLR.
- Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shum- ing Shi, and Xing Wang. 2024. Exploring human- like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229-246.
Cited by
Sorted by citations
Items 101–200 of 1,118
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
Fundamental Capabilities and Applications of Large Language Models: A Survey
Li, J., Gao, Y., Yang, Y., Bai, Y., Zhou, X., Li, Y., Sun, H., Liu, Y., Si, X., Ye, Y., et al. (2025) · ACM Computing Surveys · doi:10.1145/3735632 · Cited by 45
Self-citationXue, X., Lu, Z., Huang, D., Wang, Z., Ouyang, W., & Bai, L. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02292 · Cited by 45
Guo, X., Keivan, D., Syed, U., Qin, L., Zhang, H., Dullerud, G. E., Seiler, P. J., & Hu, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.19811 · Cited by 45
AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI CollaborationPreprint
Liu, J., Qiu, S., Li, M., Li, B., Ji, H., Han, S., Ye, X., Xia, P., Dong, Z., Chen, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.20025 · Cited by 45
Self-citationWang, Z., Zhu, Y., Zhao, H., Zheng, X., Sui, D., Wang, T., Tang, W., Wang, Y., Harrison, E. M., Pan, C., et al. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714877 · Cited by 43
Self-citationNavigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMsPreprint
Rasal, S., & Hauer, E. J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16713 · Cited by 43
Zhang, B., Mao, H., Ruan, J., Wen, Y., Li, Y., Zhang, S., Xu, Z., Li, D., Li, Z., Li, L., et al. (2026) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case69030.2026.11704361 · Cited by 41
GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
Zeng, X., Wang, X., Zhang, T., Yu, C., Zhao, S., & Chen, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3698145 · Cited by 41
Self-citationLearning to Use Tools via Cooperative and Interactive AgentsPreprint
Shi, Z., Gao, S., Chen, X., Feng, Y., Yan, L., Shi, H., Yin, D., Ren, P., Verberne, S., & Ren, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03031 · Cited by 41
Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method
Xia, T., He, Z., Ren, T., Miao, Y., Zhang, Z., Yang, Y., & Wang, R. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.213 · Cited by 40
Self-citationWhen One LLM Drools, Multi-LLM Collaboration Rules
Feng, S., Ding, W., Liu, A., Wang, Z., Shi, W., Wang, Y., Shen, Z., Han, X., Lang, H., Lee, C.-Y., et al. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.775 · Cited by 40
Self-citationThematic-LM: A LLM-based Multi-agent System for Large-scale Thematic Analysis
Qiao, T., Walker, C. G., Cunningham, C. W., & Koh, Y. S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714595 · Cited by 40
GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling
Zhou, J., Wang, L., & Yang, X. (2025) · neural information processing systems · doi:10.52202/085713-0272 · Cited by 40
Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows
Grunde-McLaughlin, M., Lam, M. S., Krishna, R., Weld, D. S., & Heer, J. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3716134 · Cited by 39
GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations
Duan, J., Zhang, R., Diffenderfer, J., Kailkhura, B., Sun, L., Stengel-Eskin, E., Bansal, M., Chen, T., & Xu, K. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-0885 · Cited by 39
Deep Researcher with Test-Time DiffusionPreprint
Han, R., Chen, Y., CuiZhu, Z., Miculicich, L., Sun, G., Bi, Y., Wen, W., Wan, H., Wen, C., Maître, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.16075 · Cited by 39
Lu, Q., Zhu, L., Xu, X., Xing, Z., Harrer, S., & Whittle, J. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00028 · Cited by 38
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
Chen, J., Prasad, A., Saha, S., Stengel-Eskin, E., & Bansal, M. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1660 · Cited by 38
Multi-expert Prompting Improves Reliability, Safety and Usefulness of Large Language Models
Long, D. X., Yen, D. N., Luu, A. T., Kawaguchi, K., Kan, M., & Chen, N. F. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1135 · Cited by 38
Reverse Thinking Makes LLMs Stronger Reasoners
Chen, J. B., Wang, Z., Palangi, H., Han, R., Ebrahimi, S., Le, L. T., Perot, V., Mishra, S. R., Bansal, M., Lee, C.-Y., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.434 · Cited by 38
Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent DebatePreprint
姚斌伟, Shang, C., Du, W., He, J., Lian, R., Zhang, Y., Su, H., Swamy, S., & Qi, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.23055 · Cited by 38
Multi-agent systems for chemical engineering: a review and perspective
Rupprecht, S., Gao, Q., Karia, T., & Schweidtmann, A. M. (2025) · Current Opinion in Chemical Engineering · doi:10.1016/j.coche.2025.101209 · Cited by 37
Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLPreprint
Sun, H., Hüyük, A., & van der Schaar, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.06553 · Cited by 37
Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred SkillsPreprint
Chen, J. C.-Y., Yun, S., Stengel-Eskin, E., Chen, T., & Bansal, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05641 · Cited by 37
Learning to break: Knowledge-enhanced reasoning in multi-agent debate system
Wang, H., Du, X., Yu, W., Chen, Q., Zhu, K., Chu, Z., Yan, L., & Guan, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2024.129063 · Cited by 36
SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
Li, D., Tan, Z., Qian, P., Li, Y., Chaudhary, K. S., Hu, L., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.03284 · Cited by 35
AUTOGEN STUDIO: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems
Dibia, V. C., Chen, J., Bansal, G., Syed, S., Fourney, A., Zhu, E., Wang, C., & Amershi, S. (2024) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2024.emnlp-demo.8 · Cited by 35
Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents
Widyasari, R., Weyssow, M., Irsan, I. C., Ang, H. W., Liauw, F., Ouh, E. L., Shar, L. K., Kang, H. J., & Lo, D. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773256 · Cited by 35
Under the Surface: Tracking the Artifactuality of LLM-Generated DataPreprint
Das, D., de Langis, K., Martin, A., Kim, J., Lee, M., Kim, Z. M., Hayati, S. A., Owan, R., Hu, B., Parkar, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.14698 · Cited by 35
Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive AnalysisPreprint
Chen, A., Song, Y., Zhu, W., Chen, K., Yang, M., Zhao, T., & Zhang, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.11544 · Cited by 35
Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided ExplorationPreprint
Gao, J., Pan, L., Wang, Y., Zhong, R., Lu, C., Wang, M., Cai, Q., Jiang, P., & Zhao, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.17621 · Cited by 35
Self-citationA Survey of Cooperative Multi-Agent Reinforcement Learning for Multi-Task Scenarios
Chai, J., Zhao, Z., Zhu, Y., & Zhao, D. (2025) · Artificial Intelligence Science and Engineering · doi:10.23919/aise.2025.000008 · Cited by 34
PhishDebate: An LLM-Based Multi-Agent Framework for Phishing Website Detection
Li, W., Manickam, S., Chong, Y.-W., & Karuppayah, S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401440 · Cited by 33
LLM Multi-Agent Systems: Challenges and Open Problems
Han, S., Zhang, Q., Jin, W., & Xu, Z. (2024) · arXiv · doi:10.48550/arxiv.2402.03578 · Cited by 33
Smart Agent-Based Modeling: On the Use of Large Language Models in Computer SimulationsPreprint
Wu, Z., Peng, R., Han, X., Zheng, S., Zhang, Y., & Xiao, C. (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2311.06330 · Cited by 33
Do LLM Agents Have Regret? A Case Study in Online Learning and GamesPreprint
Park, C., Liu, X., Ozdaglar, A., & Zhang, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.16843 · Cited by 33
Think Smarter not Harder: Adaptive Reasoning with Inference Aware OptimizationPreprint
Yu, Z., Xu, T., Jin, D., Sankararaman, K. A., He, Y., Zhou, W., Zeng, Z., Helenowski, E., Zhu, C., Wang, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.17974 · Cited by 33
Advanced Smart Contract Vulnerability Detection via LLM-Powered Multi-Agent Systems
Wei, Z., Sun, J., Sun, Y., Liu, Y., Wu, D., Zhang, Z., Zhang, X., Li, M., Liu, Y., Li, C., et al. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3597319 · Cited by 31
GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoning
Zou, H., Zhao, Q., Lasaulce, S., Bariah, L., Bennis, M., & Debbah, M. A. (2025) · IEEE Access · doi:10.1109/access.2025.3565859 · Cited by 31
Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects
Liu, Y., Zhang, G., Wang, K., Li, S., Pan, S., & An, B. (2026) · IEEE Intelligent Systems · doi:10.1109/mis.2025.3642667 · Cited by 31
Chen, J. C.-Y., Saha, S., Stengel-Eskin, E., & Bansal, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01620 · Cited by 31
Lee, S., Hwang, S., Kim, D.-J., & Lee, K. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719792 · Cited by 30
Liu, Y., Liu, Y., Zhang, X., Chen, X., & Yan, R. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730092 · Cited by 30
Risk Analysis Techniques for Governed LLM-based Multi-Agent SystemsPreprint
Reid, A., O’Callaghan, S., Carroll, L., & Caetano, T. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.05687 · Cited by 30
Yang, Y., Ma, Y., Feng, H., Cheng, Y., & Han, Z. (2025) · Applied Sciences · doi:10.3390/app15073676 · Cited by 29
Self-citationDoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2023) · arXiv · doi:10.48550/arxiv.2309.03883 · Cited by 29
Let Models Speak Ciphers: Multiagent Debate through EmbeddingsPreprint
Pham, C., Liu, B., Yang, Y., Chen, Z., Liu, T., Yuan, J., Plummer, B. A., Wang, Z., & Yang, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.06272 · Cited by 29
Self-citationFree-MAD: Consensus-Free Multi-Agent Debate
Cui, Y., Fu, H., Zhang, H., Wang, L., & Zuo, C. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1600 · Cited by 28
Empowering Scientific Workflows with Federated Agents
Kamatar, A., Pauloski, J. G., Babuji, Y., Chard, R., Sakarvadia, M., Babnigg, D., Foster, I., & Chard, K. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00114 · Cited by 28
Evaluating Persona Prompting for Question Answering Tasks
Olea, C., Tucker, H., Phelan, J., Pattison, C., Zhang, S., Lieb, M., Schmidt, D., & White, J. (2024) · International Conference on Security, Privacy and Trust Management (SPTM) · doi:10.5121/csit.2024.141106 · Cited by 28
Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon GamePreprint
Shi, Z., Fang, M., Zheng, S., Deng, S., Chen, L., & Du, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.17515 · Cited by 28
RL-GPT: Integrating Reinforcement Learning and Code-as-policyPreprint
Liu, S., Yuan, H., Hu, M., Li, Y., Chen, Y., Liu, S., Lu, Z., & Jia, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.19299 · Cited by 28
AgentCoord: Visually exploring coordination strategy for LLM-based multi-agent collaboration
Pan, B., Lu, J., Wang, K., Li, Z., Wen, Z., Feng, Y., Zhu, M., & Chen, W. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104338 · Cited by 27
Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models
Ma, J., Gao, Z., Chai, Q., Sun, W., Wang, P., Pei, H., Tao, J., Song, L., Liu, J., Zhang, C., et al. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34658 · Cited by 27
Reducing confirmation bias on controversial issues through LLM-generated multi-persona debates
Shi, L., Liu, H., Wong, Y., Mujumdar, U., Zhang, D., Jacek Gwizdka, A., & Lease, M. (2026) · Information Processing & Management · doi:10.48550/arxiv.2412.04629 · Cited by 27
Leveraging Large Language Models for Collective Decision-Making
Papachristou, M., Yang, L., & Hsu, C.-C. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757418 · Cited by 27
Thought Communication in Multiagent Collaboration
Zheng, Y., Zhao, Z., Li, Z., Xie, Y., Gao, M., Zhang, L., & Zhang, K. (2025) · neural information processing systems · doi:10.52202/085713-4113 · Cited by 27
Pan, Q., Ji, W., Ding, Y., Li, J., Chen, S., Wang, J., Zhou, J., Chen, Q., Zhang, M., Wu, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.02665 · Cited by 27
CoMAS: Co-Evolving Multi-Agent Systems via Interaction RewardsPreprint
Xue, X., Zhou, Y., Zhang, G., Zhang, Z., Li, Y., Zhang, C., Yin, Z., Torr, P. H. S., Ouyang, W., & Bai, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.08529 · Cited by 27
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26
Self-Prompt Tuning: Enable Autonomous Role-Playing in LLMs
Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Zhou, J., & Sun, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_33 · Cited by 26
Towards Rationality in Language and Multimodal Agents: A Survey
Jiang, B., Xie, Y., Wang, X., Yuan, Y., Hao, Z., Bai, X., Su, W., Taylor, C. J., & Mallick, T. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.186 · Cited by 26
Self-citationSailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from RewardsPreprint
Wu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.02686 · Cited by 26
CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent FailuresPreprint
Bonagiri, A., Borkar, D., Anderias, G. J., Rafatirad, S., & Homayoun, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.25338 · Cited by 26
Croxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M., Wills, G., Chen, G., Dligach, D., et al. (2025) · doi:10.1101/2025.04.22.25326219 · Cited by 25
DelphiAgent: A trustworthy multi-agent verification framework for automated fact verification
Xiong, C. C., Zheng, G., Ma, X., Li, C., & Zeng, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104241 · Cited by 25
Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts
Huang, T. Y., Yu, C., Shi, W., Peng, Z., Yang, D. C., Sun, W., & Shi, Y. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3716132 · Cited by 25
AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent SystemsPreprint
Dibia, V. C., Chen, J., Bansal, G., Syed, S., Fourney, A., Zhu, E., Wang, C., & Amershi, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.15247 · Cited by 25
ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM CollaborationPreprint
Estornell, A., Ton, J.-F., Yao, Y., & Yang, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.00053 · Cited by 25
Position: Towards a Responsible LLM-empowered Multi-Agent SystemsPreprint
Hu, J., Dong, Y., Ao, S., Li, Z., Wang, B., Singh, L., Cheng, G., Ramchurn, S. D., & Huang, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.01714 · Cited by 25
Coordinated LLM multi-agent systems for collaborative question-answer generation
Saadaoui, S., & Alonso, E. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114627 · Cited by 24
Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
Sun, X., Li, J., Zhong, Y., Zhao, D., & Yan, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889448 · Cited by 24
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Huang, M., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2309.17452 · Cited by 24
Self-citationCombining Multiple Large Language Models Improves Diagnostic Accuracy
Barabucci, G., Shia, V., Chu, E., Harack, B., Laskowski, K., & Fu, N. (2024) · NEJM AI · doi:10.1056/aics2400502 · Cited by 24
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 24
SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?Preprint
Yao, J., Wang, K., Hsieh, R., Zhou, H., Zou, T., Cheng, Z., Wang, Z., & Viswanath, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.12349 · Cited by 24
An Empirical Study of Group Conformity in Multi-Agent Systems
Choi, M. C., Kim, K., Chae, S., & Baek, S. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.265 · Cited by 23
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Pagliardini, M., Mohtashami, A., Fleuret, F., & Jaggi, M. (2024) · neural information processing systems · doi:10.52202/079017-4336 · Cited by 23
Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute
Chen, J., Xun, Z., Zhou, B., Qi, H., Zhang, H., Zhang, Q., chen, Y., Hu, W., Qu, Y., & Hu, S. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i24.39094 · Cited by 23
MCRanker: Generating Diverse Criteria On-the-Fly to Improve Pointwise LLM Rankers
Guo, F., Li, W., Zhuang, H., Luo, Y., Li, Y., Yan, L. W., Zhu, Q., & Zhang, Y. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703583 · Cited by 23
Training Language Models to Critique With Multi-agent Feedback
Lan, T., Zhang, W., Lyu, C., Li, S., Chen, X., Huang, H., Lin, D., Mao, X.-L., & Chen, K. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.78 · Cited by 23
LLM-Empowered State Representation for Reinforcement LearningPreprint
Wang, B., Qu, Y. P., Jiang, Y., Shao, J., Liu, C., Yang, W., & Ji, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.13237 · Cited by 23
Bridging the linguistic divide: a survey on leveraging large language models for machine translation
Gain, B., Bandyopadhyay, D., Ekbal, A., & Singh, T. N. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09919-7 · Cited by 23
A Comprehensive Survey on Large-Language-Model-Based Agents for Education
Yang, J., Wang, M., Du, X., & Na, R. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3617909 · Cited by 22
WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image AnalysisPreprint
Lyu, X., Liang, Y., Chen, W., Ding, M., Yang, J., Huang, G., Zhang, D., He, X., & Shen, L. (2025) · arXiv (Cornell University) · Cited by 22
Goridkov, N., Wang, Y., & Goucher-Lambert, K. (2024) · Procedia CIRP · doi:10.1016/j.procir.2024.01.131 · Cited by 21
Self-citationMitigating LLM Hallucinations Using a Multi-Agent Framework
Darwish, A. M., Rashed, E. A., & Khoriba, G. A. (2025) · Information · doi:10.3390/info16070517 · Cited by 21
Aegis: Automated Error Generation and Attribution for Multi-Agent SystemsPreprint
Kong, F., Zhang, R., Yin, H., Zhang, G., Zhang, X., Chen, Z., Zhang, Z., Zhang, X., Zhu, S. C., & Feng, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.14295 · Cited by 21
Mi, Z., Zheng, R., Zhong, H., Sun, Y., Kneeland, S., Moitra, S., Kutzer, K., Xu, Z., & Huang, S. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.19780.74884 · Cited by 21
Qi, M., Zhu, T., Zhang, L., Li, N., Tan, Y., & Zhou, W. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3659486 · Cited by 20
Qi, J., Li, M., Liu, J., Shu, Y., Yu, D. S. F., Ma, S., Cui, W., Zhao, Y., Chen, Y., Jiang, R., et al. (2026) · Academia AI and Applications · doi:10.20935/acadai8260 · Cited by 20
Improving Multi-Agent Debate with Sparse Communication Topology
Li, Y., Du, Y., Zhang, J., Hou, L., Grabowski, P., Li, Y., & Ie, E. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.427 · Cited by 20
Chen, A., Lou, L., Chen, K., Bai, X., Xiang, Y., Yang, M., Zhao, T., & Zhang, M. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-short.64 · Cited by 20
Learning to Use Tools via Cooperative and Interactive Agents
Shi, Z., Gao, S., Chen, X., Feng, Y., Yan, L., Shi, H., Yin, D., Ren, P., Verberne, S., & Ren, Z. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.624 · Cited by 20
Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery
Shen, C., Chen, Z., Luo, D., Xu, D., Chen, H., & Ni, J. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.36 · Cited by 20
AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System NeedPreprint
Gu, Z., Zhu, X., Cai, Y., Shen, H. F., Chen, X., Wang, Q., Li, J., Shi, X., Guo, H., Huang, W., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.15451 · Cited by 20
Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure TrajectoriesPreprint
Shao, S., Zhang, K., Li, Q., Wang, S., Wang, H., Jiao, W., Yuan, L., Guo, Y., Liu, W., & Zhang, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.02276 · Cited by 20
Self-citationEnhancing human-centered dynamic scene understanding via multiple LLMs collaborated reasoning
Zhang, H., Zhang, W., Qu, H., & Liu, J. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00074-1 · Cited by 19
Metadata
- Authors
- Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Tencent (China) (CN)
- Tsinghua University (CN)
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 17889-17904
- OpenAlex
- W4404782209
- Cited by
- 1,062
- References
- 14
- Citations by year
- 2026: 473 · 2025: 405 · 2024: 155
- FWCI
- 162.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, multi-agent debate, divergent thinking, self-reflection, tit-for-tat, LLM-as-a-judge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
