The DOI matches its DataCite record; 47 of the 48 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Open accessAbstract
Large language models (LLMs) have demonstrated remarkable capabilities in language generation, understanding, and few-shot learning in recent years. An extensive body of work has explored how their performance may be further improved through the tools of prompting, ranging from verification, self-consistency, or intermediate scratchpads. In this paper, we present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate. Overall, our findings suggest that such "society of minds" approach has the potential to significantly advance the capabilities of LLMs and pave the way for further breakthroughs in language generation and understanding.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Multiagent debate, where multiple LLM instances propose answers and critique each other over rounds, improves reasoning and factual accuracy over single-agent, reflection, and majority-vote baselines across six tasks. The approach uses black-box models and identical prompts, though it is more computationally expensive and can converge confidently on incorrect answers.
Key findings
- Debate improved reasoning: arithmetic accuracy was 81.8 ± 2.3 vs 67.0 ± 4.7 for single agent, 72.1 ± 4.5 for reflection, and 69.0 ± 4.6 for majority voting; GSM8K was 85.0 ± 3.5 vs 77.0 ± 4.2, 75.0 ± 4.3, and 81.0 ± 3.9; chess pawn score was 122.9 ± 7.6 vs 91.4 ± 10.6, 102.1 ± 11.9, and 102.2 ± 6.2.
- Debate improved factual validity: biographies accuracy was 73.8 ± 2.3 vs 66.0 ± 2.2 for single agent and 68.3 ± 2.9 for reflection; MMLU was 71.1 ± 4.6 vs 63.9 ± 4.8 and 57.7 ± 5.0; chess move validity was 45.2 ± 2.9 vs 29.3 ± 2.6 and 38.8 ± 2.9.
- On 20 GSM8K problems debated between ChatGPT and Bard, Bard solved 11, ChatGPT solved 14, and joint multi-agent debate solved 17.
- Performance increased with more agents and more debate rounds; additional rounds above four gave similar final performance to 4 rounds. Longer debate prompts led to slower convergence but better final consensus, and different persona prompts improved MMLU from 71.1 to 74.2.
Methods
- Multiple language model instances independently answer a query, then each reads and critiques the other agents’ responses and updates its own answer over multiple rounds; the same procedure and prompts are used across tasks, requiring only black-box model access.
- Evaluated on six tasks: arithmetic (100 generated tasks), GSM8K (100 problems), chess move prediction (300 games), biographies (524 well-known computer scientists), MMLU (100 questions), and chess move validity (100 tasks), using gpt-3.5-turbo-0301; main debate runs used 3 agents and 2 rounds.
- Compared against single-agent, reflection, and majority-voting baselines; reported accuracy, Stockfish pawn score, and factual validity, with ablations on number of agents, rounds, prompt length, summarization, and different models/personas.
Limitations
- Stated: multiagent debate is more computationally expensive because it requires multiple generations and debate rounds; longer debates sometimes exceed models’ processing, with models focusing mainly on recent generations; debates converge but not necessarily to correct answers, and models can confidently affirm incorrect answers because they do not correctly express uncertainty.
- Stated: the biography evaluation metric may miss incorrect generated facts not captured in the ground-truth bullets.
- Evident from the text: main evaluations often use 100 examples per task and 3 agents with 2 rounds due to computational expense, and the cross-model comparison uses only 20 GSM8K problems.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
44 Own result · 11 No basis given · 10 Own interpretation · 4 Own method · 2 Sources not found · 1 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2: Multiagent Debate Improves Factual Accuracy Multi-agent debate improves the factual accuracy.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Model Biographies MMLU Chess Move Validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate results are also less likely to include false facts that models are internally uncertain of.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we found that additional debate rounds above four led to a similar final performance to 4 rounds of debate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On arithmetic, performance monotonically increases with the increased number of agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In comparison to other prompting techniques, our multiagent debate procedure is more computationally expensive, as it requires both multiple language generations, and an underlying debate procedure.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After debating and examining the responses of other model instances, we find that the population almost always converges on a single and more accurate common answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, directly asking each agent about their confidence [10] of the answer led to high confidence assessments on each answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, these facts often inconsistent across different language model instances.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preliminary testing, we found that existing language models had a tendency to hallucinate many facts on this task.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 3.2 Extracting Factual Information from Multiagent Debate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we found that on facts that the language model was confident in (i.e.many instances of the same model all gave the same answer), it was very difficult to convince an agent to change their opinion, suggesting that "ease of persuasion" may be a method to assess factual confidence.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 9
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this set, we find that multi-agent debate improves the performance of both agents, with Bard solving 11 problems, chatGPT solving 14 problems, and joint multi-agent debate solving 17 problems.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using both multiple model agents and multiple rounds of debate are important to achieve the best performance.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because as the debate progresses, individual model instances tend to disagree on uncertain facts and omit them from the answer (Figure 7).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both settings, multiagent generation is beneficial.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the approach is simple and effective across a wide set of different reasoning and validity language modeling tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 5 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that multiagent debate improved and settled on bullets that were more consistent across agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although we found further gains with both more agents and rounds of debate (Figure 10).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we found that while debates typically converged into single final answers, these answers were not necessarily correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While both agents initially provide incorrect answers to the problem, chatGPT is able to utilize the incorrect response given by Bard to generate the final correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Overall, we observed that language model agents were relatively "agreeable"”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that prompts that encouraged models to be more "stubborn' based on their own solutions led to longer debates and better final solutions.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, we observed that as debates became longer in duration, current language models sometimes struggled to fully process the entire debate input, and typically only focused on the most recent generations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that debates using longer prompts lead to slower convergence to correct answers, but also lead to a better final consensus on the correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Effect of Debate Length on Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, debate gives the best performance in this setting also, and significantly outperforms each baseline.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When there are many agents in a debate, responses from other agents may be first summarized and then given as context, reducing context length.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate causes agents to converge to one fact that is more accurate, but not necessarily always factually correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we found that generated biographies after debate to be more accurate, many facts remain incorrect.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that debate improved the performance of final generated answers, though sometimes answers would converge to the incorrect value.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional Qualitative Visualizations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using a short debate prompt induces faster consensus between agents”
- Also stated in: § Agent 1: · plain
“Using a short debate prompt induces faster consensus between agents”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > A Appendix
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Agent 1: · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The method is also orthogonal to other model generation improvements such as retrieval or prompt engineering (in fact, we combine our debate method with zero-shot chain of thought).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to computational expense, we evaluate our approach across benchmarks mainly using three agents with two rounds of debates”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As the focus of our experiments is to verify the effectiveness of multiagent agent debate, we run both baselines and our approach, using the identical starting prompt and language model across all evaluations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the purpose of our debate isn't just to amplify a correct answer -all models can initially be wrong but arrive at the correct answer through the debate process.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting further gains can be obtained with different initialization prompts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.”
- Also stated in: § 1 Introduction · plain
“We use the same methodology and prompt templates for all our tasks and require only black-box access to language model generations -no model-internal information such as likelihoods or gradients is needed.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments we use the same prompts for all agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
“We constructed ground truth bullet point biographies of 524 well-known computer scientists.”
The paper relies on this without support.
The chatGPT-based language model used in the experiments is representative of contemporary language models.
“We use chatGPT-based language model [21] in all our experiments except those in Figure 11 where we compare multiple language models.”
The paper relies on this without support.
Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
“We then asked language models to generate bullet point biographies for each person, and evaluated the accuracy at which each ground truth bullet point agreed with generated bullets.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this method, the capability of each agent is enhanced by understanding and incorporating the other agents' opinions.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“consisting of six different two-digit numbers”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“For example: What is the result of 12+15*21+0-3*27?”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“0.5 × 3 × 4 = 64”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.1 Multiagent Language Generation
Reported 64 · recomputed 6
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
“uses identical procedure and prompts for all tasks we investigate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Arithmetic
“## Arithmetic Starting What is the result of {}+{}*{}+{}-{}*{}?Make sure to state your answer at the end of the response.”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“To improve reasoning, approaches have relied on prompting techniques such as scratchpads [20], verification [3], chain-of-thought demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and finetuning [13, 24, 31].”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Overall, we observed that language model agents were relatively "agreeable", perhaps as a result of instruction tuning or reinforcement learning based on human feedback (Ouyang et al., 2022).”
Minsky, M. (1987) · Personalist Forum · doi:10.5840/persforum19873112 · Cited by 2,415 · cited 2×
“Instead, we propose a complementary approach inspired by The Society of Mind (Minsky, 1988) and multiagent settings, where multiple language model instances (or agents) individually propose and jointly debate their responses and reasoning processes to arrive at a common answer.”
InfluentialSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 2×
“We find that our debate approach outperforms single model baselines such as zero-shot chain of thought (Kojima et al., 2022) and reflection (Shinn et al., 2023; Madaan et al., 2023) on a variety of six reasoning, factuality, and question-answering tasks.”
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3×
“In Figure 6, we illustrate the performance of multi-agent debate with and without zero-shot chain-ofthought prompting [11] on GSM8K.”
BackgroundMethodLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Research works printed but not linked: 1
- Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 9
Cited by
Sorted by citations
Items 101–181 of 181
LLM-SAP: Large Language Models Situational Awareness-Based Planning
Wang, L., & Zhong, H. (2024) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew63481.2024.10645429 · Cited by 7
“Such collaboration leverages the individual strengths of each agent [28, 29, 30].”
BackgroundBeyond Isolation: Multi-agent Synergy for Improving Knowledge Graph Construction
Ye, H., Gui, H., Zhang, A., Liu, T., & Jia, W. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-1809-5_6 · Cited by 17
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.335 · Cited by 4
Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
de Cerqueira, J. A. S., Agbese, M., Rousi, R., Xi, N., Hamari, J., & Abrahamsson, P. (2024) · arXiv · doi:10.48550/arxiv.2411.08881 · Cited by 4
“The aforementioned studies run in line with a joint study by MIT and Microsoft, Du et al. [24].”
Cooperative Scheduling and Hierarchical Memory Model for Multi-Agent Systems
Zou, H., Li, R., Sun, T., Wang, F., Li, T., & Liu, K. (2024) · 2024 IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA) · doi:10.1109/ispce-asia64773.2024.10756271 · Cited by 4
Mugambiwa, T., & Ndlovu, B. (2026) · Journal of Applied Informatics and Computing · doi:10.30871/jaic.v10i1.11900 · Cited by 4
“Decomposing the reasoning process into modular steps meant that for every diagnostic or interpretive decision, clinicians could follow how it resulted[43].”
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
“As discussed in Section 5.3, weak feedback can cause over-refinement of good responses while failing to fix bad ones, negatively affecting overall gains.”
InfluentialOn the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“These take many forms: common search algorithms with the LLM cast in the role of heuristic [37, 11], approaches which reduce error rates by enforcing various consistency conditions [8, 6, 17], and direct LLM self-critique [27, 35, 5, 15, 21, 38].”
MethodSWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2410.20285 · Cited by 3
“Inspired by multi-agent debate frameworks Du et al. (2023); Khan et al. (2024); Amayuelas et al. (2024), this agent engages in a structured debate, where multiple agents argue for or against the proposed solutions.”
Language Agents as Optimizable Graphs
Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., & Schmidhuber, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16823 · Cited by 3
Single-agent or Multi-agent Systems? Why Not Both?
Gao, M., Li, Y., Liu, B., Yu, Y., Wang, P., Lin, C.-Y., & Lai, F. (2025) · arXiv · doi:10.48550/arxiv.2505.18286 · Cited by 3
Responsible Integration of Large Language Models (LLMs) in Navy Operational Plan Generation
Kapiamba, S., Fouad, H. Y., & Moskowitz, I. S. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31179 · Cited by 2
Fostering collective intelligence in CPSS: an LLM-driven multi-agent cooperative tuning framework
Chen, R., & He, C. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1613499 · Cited by 4
Application of Self-Evolving AI Agents in Chemical Research: A Novel Intelligent Assistance SystemPreprint
Ma, K. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-6tv8c · Cited by 1
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 24
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2024) · arXiv · doi:10.48550/arxiv.2410.04444 · Cited by 2
“– Explore techniques like: LLM Debate, Step-back Abstraction, Dynamic Assignment of Roles, and so on.”
InfluentialImproving Accuracy and Generalizability via Multi-Modal Large Language Models Collaboration
Zhang, S., Mu, H., & Liu, T. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651475 · Cited by 3
“Recently, a team from the Computer Science and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute of Technology [9] has introduced a strategy of utilizing multiple artificial intelligence systems to discuss and debate with each other, in order to achieve convergence of the…”
BackgroundVettermann, S., & Röper, M. (2025) · Zeitschrift für wirtschaftlichen Fabrikbetrieb · doi:10.1515/zwf-2025-0003 · Cited by 1
The superalignment of superhuman intelligence with large language models
Huang, M., Wang, Y., Cui, S., Ke, P., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4348-6 · Cited by 3
“Third, since in general purposing a stronger model is very hard to learn from weaker supervision, can we assemble multiple specialized weaker models to supervise the learning of a stronger model?”
EMRDCM : An experience-enhanced multi-role debate method based on compromise mechanisms
Zhang, Y., Feng, J., Yiming, X., Wang, Z., Yang, T., & Wang, X. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01022-1 · Cited by 1
“Given the costs associated with API-based modeling and the limitations on the number of API calls, we followed much of the previous work (Du et al., 2024; Bian et al., 2024; Besta et al., 2024; Yao et al., 2024) by randomly selecting 2,000 training samples and 500 test samples in each dataset for…”
Kraidia, I., Qaddara, I., Almutairi, A., Alzaben, N., & Belhouari, S. B. (2026) · Scientific Reports · doi:10.1038/s41598-026-42705-7 · Cited by 8
Fang, W., Li, L., Shi, K., Yang, Y., & Zhang, J. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792976 · Cited by 1
From LLM narratives to parameterized cooperation policies in multi-agent systems
de Curtò, J., de Zarzà, I., Cabot, J., Manzoni, P., & Calafate, C. T. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1820827 · Cited by 2
“…agents paradigm (Park et al., 2023) demonstrated that LLM-driven agents can exhibit emergent social behavior, and subsequent work has scaled this to multi-agent debate (Du Y. et al., 2024; Liang et al., 2024), role-playing societies (Li et al., 2024), and cooperative dilemmas (Piatti et al., 2024).”
CMMQC: Cascaded Multi-Model Quality Control for Unsupervised Data-to-Text Generation
Zhang, W., Sun, X., Luo, Y., Gao, W., & Zhu, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650876 · Cited by 1
“The ”society of minds” concept leverages model collaboration to enhance reasoning and language skills [15].”
BackgroundConfidenceCal: Enhancing LLMs Reliability through Confidence Calibration in Multi-Agent Debate
Bai, Y. (2024) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia63733.2024.10808396 · Cited by 2
A GenAI System for Improved FAIR Independent Biological Database Integration
Sakib, S. N., Naha, K., Rubaiat, S. Y., & Jamil, H. M. (2025) · Journal of Data and Information Quality · doi:10.1145/3770753 · Cited by 2
“While parts of the system are still under development, and are in flux, all functions described in this article are active.”
Kuerbanjiang, W., Wang, X., Jiamaliding, Y., Maimaitiaili, M., & Yi, Y. (2025) · medRxiv · doi:10.1101/2025.10.30.25339199 · Cited by 1
Diversity of Thought Elicits Stronger Reasoning Capabilities in Multi-Agent Debate Frameworks
University of Montreal, M. Q. A. I., & Hegazy, M. (2024) · Journal of Robotics and Automation Research · doi:10.33140/jrar.05.03.06 · Cited by 1
Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate
Wynn, A., Satija, H., & Hadfield, G. (2025) · arXiv · doi:10.48550/arxiv.2509.05396 · Cited by 96
“Various forms of multi-agent debate have been shown to improve performance on multiple arith-* Proceedings of the 42 nd International Conference on Machine Learning , Vancouver, Canada.”
InfluentialMDTeamGPT: Mitigating Context Collapse and Enabling Self-Evolution in Medical Multi-Agent Reasoning
Chen, K., Li, X., Yang, T., Wang, H., Yang, G., Huo, J., & Gao, Y. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1427 · Cited by 2
“Consequently, Multi-Agent frameworks have emerged as a promising alternative (Liu et al., 2023; Du et al., 2023), which elicit latent medical knowledge and mitigate individual model hallucinations through collaborative debate.”
Robeyns, M., Szummer, M., & Aitchison, L. (2025) · arXiv · doi:10.48550/arxiv.2504.15228 · Cited by 117
Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity
Yang, Y., Qu, C., Wen, M., Shi, L., Wen, Y., Zhang, W., Wierman, A., & Gu, S. (2026) · arXiv · doi:10.48550/arxiv.2602.03794 · Cited by 1
Do as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1
“One prominent application area is world simulation [84], where these systems employ multiple agents to replicate interactions in diverse fields such as gaming [85–89], economy [90–94], science debate [95–97], and disease [98–101].”
Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
Wu, H., Li, Z., & Li, L. (2025) · arXiv · doi:10.48550/arxiv.2511.07784 · Cited by 1
“…* These authors contribute equally to this work. they often reach higher accuracy than a single LLM, supporting the “society of minds” view (Du et al., 2023), with further demonstrated gains in mathematics (Zhang and Xiong, 2025), healthcare decision-making (Lu et al., 2024), and factual…”
InfluentialGómez Álvarez, S., Mozo Quesada, A., Navarro, T., Gálvez Rojas, S., & López Valverde, F. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01086-z · Cited by 1
Taylor, E. (2026) · Ethics & Human Research · doi:10.1002/eahr.70024
Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
Wunderlich, F. V., Kaesberg, L. B., Wahle, J. P., Ruas, T., & Gipp, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) · doi:10.18653/v1/2026.acl-srw.1 · Cited by 5
“Inference-time strategies such as self-consistency (Wang et al., 2023), self-refinement (Madaan et al., 2023), debate (Du et al., 2023), and mixture-of-agents (MoA) (Wang et al., 2024a) differ in their approach to scale inference compute, for example, through additional sampling, iterative…”
InfluentialDeng, Y., Heyan, H., Zhang, X., Hu, Y., Weng, J., & Li, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4811426
Stein, J., Romeijn, J., & Mäs, M. (2024) · doi:10.31219/osf.io/n4asf
Agent-Based Models as Digital Twins in Management Science
Mäs, M. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197668122.013.23 · Cited by 1
Stein, J., Romeijn, J., & Mäs, M. (2025) · Erkenntnis · doi:10.1007/s10670-024-00913-5
“Similarly, recent advances in the development of large-language models make it possible to formally represent argument communication in terms of realistic human language rather than the abstract representation of arguments applied here (Betz, 2022; Du et al., 2023).”
A Multi-Attribute Mixture Expert Reasoning Approach Based on Large Language ModelsPreprint
Zhang, L., Yuan, J., He, Y., Yu, M., Zhu, K., Luo, J., Yu, Z., & Li, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5167458
Jeong, M., Kim, T., Choi, K., Kim, S., Baek, S., & Kim, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5224792
Multi-Agent Collaboration for Hallucination Correction in Complaint Management
ÇELDİR, O. M., & Dalkılıç, G. (2025) · International Symposium on Innovative Approaches in Smart Technologies (ISAS) · doi:10.1109/isas66241.2025.11101785
COLLAB-RAG: Cognitive Orchestration of Latent Logic via Multi-Role Adversarial RationalisationPreprint
Zhang, K., han, X., Liu, C., Tian, Y., yuan, D., & Cheng, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5617131
SocraSynth: Adversarial Multi-LLM Reasoning
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749430
“Two other recent studies [Du et al. 2024, Khan et al. 2024] have also focused on enhancing the accuracy of LLM-generated responses.”
SagaLLM: Persistent Context Management, Constraint Validation, and Transaction Guarantees
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749435
Schnepf, J., Schwarz, M., Scheuermann, B., & Anderer, S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_7
Hallucination-Free Causal Graph-Guided AI Framework for Intuitive Question and Answer Generation
Wang, N., & Katsaggelos, A. K. (2026) · International Journal of Multimedia Data Engineering and Management · doi:10.4018/ijmdem.404003
From Code to Rubrics: A Multimodal Evaluation of LLM Systems for Automated Feedback Generation
Haseeb, M., Amjad, A. I., Amjad, M., & Sheng, V. S. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795458
“While prior work on multi-agent and role-specialized prompt-based strategy suggested that decomposing complex problems into coordinated agents could increase the reasoning and output capabilities of LLMs, especially small LLMs (sLLMs) setups with resource constraints [36].”
Wankhade, A., & Shahade, A. K. (2026) · The International journal of networked and distributed computing · doi:10.1007/s44227-026-00105-3
“Multi-agent debate frameworks [ 37 ] use adversarial cross-checking to reduce hallucinations.”
Seo, Y. I., Lee, N., Min, H., et al. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808379
“In parallel, structured multi-agent debate and schema-guided role-play show that LLMs can follow explicit protocols to produce consistent, role-specific reasoning traces [3, 5].”
Five AI Agents, One Hidden Source: The Independence Illusion in Multi-Agent Decisions
Tan, K. H. (2026) · doi:10.68050/jams.2026.489 · Cited by 1
An agentic AI environment to support biomedical research in collaborative academic environmentsPreprint
Wu, T., Browne, T. S., Meawad, M., Emam, H. E., Simsam, N. H., Xia, Y., Jaafar, A., Ma, A., Kabbani, B., Gupta, V., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.24.753565
Triarchic corrigibility: a role-separated governance design pattern for AI-assisted decision-making
Kolodziej, M. C. (2026) · AI and Ethics · doi:10.1007/s43681-026-01362-2
“Multi-model debate can improve some tasks [22], but ability imbalance and homogeneous, uncalibrated debate can produce domination or underperform simpler aggregation [ 18, 19].”
Chen, Y. (2026) · arXiv · doi:10.48550/arxiv.2606.14923 · Cited by 2
“A growing body of work places LLM agents in cooperative and competitive settings [14, 8, 18, 17], including repeated economic games [1], negotiation [13], and multi-agent debate in which agents cross-check one another [6, 10].”
Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats
Thota, P., Lei, Y., Thangaraj, S., Jonnalagadda, S. R., & Nilizadeh, S. (2026) · arXiv · doi:10.48550/arxiv.2606.26377
“Multi-agent frameworks have demonstrated strong performance in collaborative reasoning, debate, coding, and planning [28], [29], [30], [31], [32], [33].”
Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
Qiu, E. S., & Gill, J. (2026) · arXiv · doi:10.48550/arxiv.2608.18167 · Cited by 2
“MARS (Wang et al., 2025) reduces MAD-style communication costs by removing message passing between re-viewers.”
InfluentialSMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
Li, D., Tan, Z., Qian, P., Li, Y., Chaudhary, K. S., Hu, L., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.03284 · Cited by 35
“One line of work is layer-based multi-agent methods (Liang et al., 2023; Du et al., 2023; Chan et al.)”
MethodMulti-Agent Debate Strategies: Survey, Taxonomy, and Challenges
Motger, Q., Oriol, M., Marco, J., & Franch, X. (2026) · arXiv · doi:10.48550/arxiv.2607.26212 · Cited by 5
Lodhiya, H., McManus, A., & Walker, R. (2026) · arXiv · doi:10.48550/arxiv.2609.27844
Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Ferreira, L., Liu, G., & Zheng, K. (2026) · arXiv · doi:10.48550/arxiv.2609.35875
“In multi-agent debate (MAD), multiple instances of language-model agents independently propose solutions to a given task, read one another’s reasoning, and iteratively refine their answers; Du et al. (2023) report significant gains over single-agent inference for GPT-3.”
InfluentialMulti-agent discussion gains less when dissent is withheld
Mansuri, C. S., Wang, X., Li, M., Acton, B., Eckardt, R., Patel, D., & Kojaku, S. (2026) · arXiv · doi:10.48550/arxiv.2609.38324 · Cited by 1
“In contrast, other studies report that discussion can outperform individual reasoning or voting (Du et al., 2024; Liang et al., 2024; Chen et al., 2024).”
InfluentialMeasuring Collapse and Correction in Homogeneous-Panel LLM Debate
Li, X., Liu, M., & Yuen, C. (2026) · arXiv · doi:10.48550/arxiv.2609.35279
“We use a standard debate protocol [7]: three agents answer independently (Round 0 ), then discuss and update for Rounds 1 – 3 while seeing all current answers.”
Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
Li, W., Lin, Y., Xia, M., & Jin, C. (2025) · arXiv · doi:10.48550/arxiv.2502.00674 · Cited by 62
Shen, X., Liu, Y., Dai, Y., Wang, Y., Miao, R., Tan, Y., Pan, S., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2505.23352 · Cited by 6
“…al., 2023); 2) multi-agent systems with fixed topologies including Chain, Tree, Complete Graph, and Random Graph (Qian et al., 2025); 3) LLM-Debate (Du et al., 2023); and 4) automated design approaches including Agent-Prune (Zhang et al., 2025a), AgentDrop (Wang et al., 2025), and G-Designer…”
InfluentialHeterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Feng, S., Wang, Z., Goyal, P., Wang, Y., Shi, W., Xia, H., Palangi, H., Zettlemoyer, L., Tsvetkov, Y., Lee, C.-Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04510 · Cited by 18
“Role-based approaches typically rely on assigning roles to LLMs through prompt engineering [21, 25].”
InfluentialRCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory
Liu, J., Kong, Z., Yang, C., Yang, F., Li, T., Dong, P., Nanjekye, J., Tang, H., Yuan, G., Niu, W., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.04903
“For RCR-router, we evaluate both the one-shot setting ( K = 1 ; each agent is invoked once per round) and the iterative routing setting ( K > 1 ; agents reason and revise with updated context across multiple sub-steps).”
At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent
Dylan, D., Brennan, A., Murphy, C., O'Sullivan, N., Kelly, C., & Walsh, S. (2026) · arXiv · doi:10.48550/arxiv.2609.04217 · Cited by 3
Multi-agent Architecture Search via Agentic Supernet
Zhang, G., Niu, L., Fang, J., Wang, K., Bai, L., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2502.04180
“The agentic supernet can be seen as a cascaded multi-layer workflow, including ❶ multiple agentic operators ( e.g. , CoT (Wei et al., 2022), Multi-agent Debate (Du et al., 2023), ReAct (Yao et al., 2023)), as well as ❷ the parameterized probability distributions of operators across layers.”
InfluentialMAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
Ke, Z., Xu, A., Ming, Y., Nguyen, X.-P., Chin, R., Xiong, C., & Joty, S. (2025) · arXiv · doi:10.48550/arxiv.2505.14996
“Following [10], we generate multiple feedback candidates (three in our experiments) from the meta-agent and then use an additional call to the meta-agent to select the best one.”
EvoFlow: Evolving Diverse Agentic Workflows On The Fly
Zhang, G., Chen, K., Wan, G., Chang, H., Cheng, H., Wang, K., Hu, S., & Bai, L. (2025) · arXiv · doi:10.48550/arxiv.2502.07373
“…workflows , including Chain-of-Thought (Wei et al., 2022), ComplexCoT (Fu et al., 2022)), Self-Consistency (SC) (Wang et al., 2023a), LLM-Debate (Du et al., 2023), LLM-Blender (Jiang et al., 2023), DyLAN (Liu et al., 2023), AgentVerse (Chen et al., 2023d) and MacNet (Qian et al., 2024); (2)…”
InfluentialMS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios
Ding, B., Wang, Q., Ruan, L., Lu, W., Ge, Q., Fang, R., & Lin, T. (2026) · arXiv · doi:10.48550/arxiv.2610.22869
Decentralized Multi-Agent Systems with Shared Context
Mao, Y., Gu, J., Chauhan, A., Zhang, Q., Kang, H., & Mirhoseini, A. (2026) · arXiv · doi:10.48550/arxiv.2606.10662
“Instead, D E LM uses the verified shared context as a common communication substrate: once an update is admitted, it becomes visible to all agents as reusable problem state.”
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Liu, Y., James, J., Wang, Y., Zhao, K., Xiao, C., & Lin, C. (2026) · arXiv · doi:10.48550/arxiv.2610.07109
“Debate-style approaches let agents exchange rationales before producing a final answer (Chan et al., 2024; Du et al., 2024); jury-style approaches aggregate judgments from several models without explicit multi-turn interaction (Verga et al., 2024; Zhao et al., 2025b); and ContrastScore uses…”
GitSwarm: Decentralized Compounding Inference
Shah, V., Samanta, A., Dahal, P., Plekhanov, M., Wu, C.-J., Yih, S., Munos, R., Fergus, R., Foerster, J., Salakhutdinov, R., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.04862
Emergence of Biased Consensus in Multi-Agent LLM Debates
Okawa, M. (2026) · arXiv · doi:10.48550/arxiv.2608.02827
“Multi-agent LLM debate (Du et al., 2023; Liang et al., 2023; Chen et al., 2024) marks a shift from single model inference toward deliberation through collaboration of multiple LLMs due to their performance gains.”
Zhang, Y., Liu, F., Shan, Y., Huang, X., Yang, X., Zhu, Y., Cheng, X., Liu, C., Zeng, K., Zhang, T. J., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.01045
“Other efforts, such as debate-based systems (Du et al., 2023) or Mixture-of-Agents (Wang et al., 2024a), often prescribe static topological constraints that limit dynamic information flow.”
AgentScope: A Flexible yet Robust Multi-Agent Platform
Gao, D., Li, Z., Pan, X., Kuang, W., Ma, Z., Qian, B., Wei, F., Zhang, W., Xie, Y., Chen, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.14034 · Cited by 11
“The proxy of the user.”
Metadata
- Authors
- Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.14325
- OpenAlex
- W4378509427
- Cited by
- 1,202
- References
- 48
- Citations by year
- 2026: 547 · 2025: 340 · 2024: 258 · 2023: 56
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, factuality, hallucination mitigation, mathematical reasoning, strategic reasoning, Society of Mind, self-consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
