The DOI matches its DataCite record; 47 of the 48 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Open accessAbstract
Large language models (LLMs) have demonstrated remarkable capabilities in language generation, understanding, and few-shot learning in recent years. An extensive body of work has explored how their performance may be further improved through the tools of prompting, ranging from verification, self-consistency, or intermediate scratchpads. In this paper, we present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate. Overall, our findings suggest that such "society of minds" approach has the potential to significantly advance the capabilities of LLMs and pave the way for further breakthroughs in language generation and understanding.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Multiagent debate, where multiple LLM instances propose answers and critique each other over rounds, improves reasoning and factual accuracy over single-agent, reflection, and majority-vote baselines across six tasks. The approach uses black-box models and identical prompts, though it is more computationally expensive and can converge confidently on incorrect answers.
Key findings
- Debate improved reasoning: arithmetic accuracy was 81.8 ± 2.3 vs 67.0 ± 4.7 for single agent, 72.1 ± 4.5 for reflection, and 69.0 ± 4.6 for majority voting; GSM8K was 85.0 ± 3.5 vs 77.0 ± 4.2, 75.0 ± 4.3, and 81.0 ± 3.9; chess pawn score was 122.9 ± 7.6 vs 91.4 ± 10.6, 102.1 ± 11.9, and 102.2 ± 6.2.
- Debate improved factual validity: biographies accuracy was 73.8 ± 2.3 vs 66.0 ± 2.2 for single agent and 68.3 ± 2.9 for reflection; MMLU was 71.1 ± 4.6 vs 63.9 ± 4.8 and 57.7 ± 5.0; chess move validity was 45.2 ± 2.9 vs 29.3 ± 2.6 and 38.8 ± 2.9.
- On 20 GSM8K problems debated between ChatGPT and Bard, Bard solved 11, ChatGPT solved 14, and joint multi-agent debate solved 17.
- Performance increased with more agents and more debate rounds; additional rounds above four gave similar final performance to 4 rounds. Longer debate prompts led to slower convergence but better final consensus, and different persona prompts improved MMLU from 71.1 to 74.2.
Methods
- Multiple language model instances independently answer a query, then each reads and critiques the other agents’ responses and updates its own answer over multiple rounds; the same procedure and prompts are used across tasks, requiring only black-box model access.
- Evaluated on six tasks: arithmetic (100 generated tasks), GSM8K (100 problems), chess move prediction (300 games), biographies (524 well-known computer scientists), MMLU (100 questions), and chess move validity (100 tasks), using gpt-3.5-turbo-0301; main debate runs used 3 agents and 2 rounds.
- Compared against single-agent, reflection, and majority-voting baselines; reported accuracy, Stockfish pawn score, and factual validity, with ablations on number of agents, rounds, prompt length, summarization, and different models/personas.
Limitations
- Stated: multiagent debate is more computationally expensive because it requires multiple generations and debate rounds; longer debates sometimes exceed models’ processing, with models focusing mainly on recent generations; debates converge but not necessarily to correct answers, and models can confidently affirm incorrect answers because they do not correctly express uncertainty.
- Stated: the biography evaluation metric may miss incorrect generated facts not captured in the ground-truth bullets.
- Evident from the text: main evaluations often use 100 examples per task and 3 agents with 2 rounds due to computational expense, and the cross-model comparison uses only 20 GSM8K problems.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
44 Own result · 11 No basis given · 10 Own interpretation · 4 Own method · 2 Sources not found · 1 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2: Multiagent Debate Improves Factual Accuracy Multi-agent debate improves the factual accuracy.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Model Biographies MMLU Chess Move Validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate results are also less likely to include false facts that models are internally uncertain of.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we found that additional debate rounds above four led to a similar final performance to 4 rounds of debate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On arithmetic, performance monotonically increases with the increased number of agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In comparison to other prompting techniques, our multiagent debate procedure is more computationally expensive, as it requires both multiple language generations, and an underlying debate procedure.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After debating and examining the responses of other model instances, we find that the population almost always converges on a single and more accurate common answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, directly asking each agent about their confidence [10] of the answer led to high confidence assessments on each answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, these facts often inconsistent across different language model instances.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preliminary testing, we found that existing language models had a tendency to hallucinate many facts on this task.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 3.2 Extracting Factual Information from Multiagent Debate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we found that on facts that the language model was confident in (i.e.many instances of the same model all gave the same answer), it was very difficult to convince an agent to change their opinion, suggesting that "ease of persuasion" may be a method to assess factual confidence.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 9
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this set, we find that multi-agent debate improves the performance of both agents, with Bard solving 11 problems, chatGPT solving 14 problems, and joint multi-agent debate solving 17 problems.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using both multiple model agents and multiple rounds of debate are important to achieve the best performance.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because as the debate progresses, individual model instances tend to disagree on uncertain facts and omit them from the answer (Figure 7).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both settings, multiagent generation is beneficial.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the approach is simple and effective across a wide set of different reasoning and validity language modeling tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 5 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that multiagent debate improved and settled on bullets that were more consistent across agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although we found further gains with both more agents and rounds of debate (Figure 10).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we found that while debates typically converged into single final answers, these answers were not necessarily correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While both agents initially provide incorrect answers to the problem, chatGPT is able to utilize the incorrect response given by Bard to generate the final correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Overall, we observed that language model agents were relatively "agreeable"”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that prompts that encouraged models to be more "stubborn' based on their own solutions led to longer debates and better final solutions.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, we observed that as debates became longer in duration, current language models sometimes struggled to fully process the entire debate input, and typically only focused on the most recent generations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that debates using longer prompts lead to slower convergence to correct answers, but also lead to a better final consensus on the correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Effect of Debate Length on Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, debate gives the best performance in this setting also, and significantly outperforms each baseline.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When there are many agents in a debate, responses from other agents may be first summarized and then given as context, reducing context length.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate causes agents to converge to one fact that is more accurate, but not necessarily always factually correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we found that generated biographies after debate to be more accurate, many facts remain incorrect.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that debate improved the performance of final generated answers, though sometimes answers would converge to the incorrect value.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional Qualitative Visualizations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using a short debate prompt induces faster consensus between agents”
- Also stated in: § Agent 1: · plain
“Using a short debate prompt induces faster consensus between agents”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > A Appendix
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Agent 1: · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The method is also orthogonal to other model generation improvements such as retrieval or prompt engineering (in fact, we combine our debate method with zero-shot chain of thought).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to computational expense, we evaluate our approach across benchmarks mainly using three agents with two rounds of debates”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As the focus of our experiments is to verify the effectiveness of multiagent agent debate, we run both baselines and our approach, using the identical starting prompt and language model across all evaluations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the purpose of our debate isn't just to amplify a correct answer -all models can initially be wrong but arrive at the correct answer through the debate process.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting further gains can be obtained with different initialization prompts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.”
- Also stated in: § 1 Introduction · plain
“We use the same methodology and prompt templates for all our tasks and require only black-box access to language model generations -no model-internal information such as likelihoods or gradients is needed.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments we use the same prompts for all agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
“We constructed ground truth bullet point biographies of 524 well-known computer scientists.”
The paper relies on this without support.
The chatGPT-based language model used in the experiments is representative of contemporary language models.
“We use chatGPT-based language model [21] in all our experiments except those in Figure 11 where we compare multiple language models.”
The paper relies on this without support.
Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
“We then asked language models to generate bullet point biographies for each person, and evaluated the accuracy at which each ground truth bullet point agreed with generated bullets.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this method, the capability of each agent is enhanced by understanding and incorporating the other agents' opinions.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“consisting of six different two-digit numbers”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“For example: What is the result of 12+15*21+0-3*27?”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“0.5 × 3 × 4 = 64”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.1 Multiagent Language Generation
Reported 64 · recomputed 6
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
“uses identical procedure and prompts for all tasks we investigate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Arithmetic
“## Arithmetic Starting What is the result of {}+{}*{}+{}-{}*{}?Make sure to state your answer at the end of the response.”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“To improve reasoning, approaches have relied on prompting techniques such as scratchpads [20], verification [3], chain-of-thought demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and finetuning [13, 24, 31].”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Overall, we observed that language model agents were relatively "agreeable", perhaps as a result of instruction tuning or reinforcement learning based on human feedback (Ouyang et al., 2022).”
Minsky, M. (1987) · Personalist Forum · doi:10.5840/persforum19873112 · Cited by 2,415 · cited 2×
“Instead, we propose a complementary approach inspired by The Society of Mind (Minsky, 1988) and multiagent settings, where multiple language model instances (or agents) individually propose and jointly debate their responses and reasoning processes to arrive at a common answer.”
InfluentialSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 2×
“We find that our debate approach outperforms single model baselines such as zero-shot chain of thought (Kojima et al., 2022) and reflection (Shinn et al., 2023; Madaan et al., 2023) on a variety of six reasoning, factuality, and question-answering tasks.”
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3×
“In Figure 6, we illustrate the performance of multi-agent debate with and without zero-shot chain-ofthought prompting [11] on GSM8K.”
BackgroundMethodLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Research works printed but not linked: 1
- Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 9
Cited by
Sorted by citations
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 1,062
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“For example, Du et al. [92] introduced the concept of debate, endowing agents with responses from fellow peers.”
InfluentialChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
“Even if the possibility of LLM hallucination is deemed to be likely in certain assessments, recent studies demonstrate that with careful strategies and prompting techniques, hallucination can to a large degree be mitigated, though not entirely eliminated [22,50–53].”
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“This feedback typically comes from external knowledge sources through retrieval processes (He et al., 2023; Gou et al., 2023), interactive debates with other LLMs (Du et al., 2023; Cohen et al., 2023), or guidance from external evaluation metrics (Lei et al., 2023; Khalifa et al., 2023).”
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Mandi, Z., Jain, S., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.04738 · Cited by 180
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Mandi, Z., Jain, S., & Song, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610855 · Cited by 316
“Recent work also shows a dialog or debate style prompting can improve LLMs’ performance on human alignment [16] and a broad range of goal-oriented tasks [31], [25], [12].”
BackgroundFrom text to insight: large language models for chemical data extraction
Schilling-Wilhelmi, M., Ríos-García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Marquez, J. A., & Jablonka, K. M. (2024) · Chemical Society Reviews · doi:10.1039/d4cs00913d · Cited by 178
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01274 · Cited by 270
“Researches on mitigating current LLMs’ hallucination issues often focuses on several aspects, including refining the training process, us-ing larger and more diverse datasets [24], or implementing post-training evaluation [11] and correction mechanisms [33, 34].”
Large Generative AI Models for Telecom: The Next Big Thing?
Bariah, L., Zhao, Q., Zou, H., Tian, Y. T., Bader, F., & Debbah, M. A. (2024) · IEEE Communications Magazine · doi:10.1109/mcom.001.2300364 · Cited by 142
LILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Jiang, Z., Liu, J., Chen, Z., Li, Y., Huang, J., Huo, Y., He, P., Gu, J., & Lyu, M. R.-T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643733 · Cited by 148
“Therefore, compared to traditional parsing tools, the overhead of querying LLMs ( e.g. , inference time and network latency) is notably high [10, 58].”
Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity
Novelli, C., Casolari, F., Hacker, P., Spedicato, G., & Floridi, L. (2024) · Computer Law & Security Review · doi:10.1016/j.clsr.2024.106066 · Cited by 124
How large language models can reshape collective intelligence
Burton, J. W., Lopez-Lopez, E., Hechtlinger, S., Rahwan, Z., Aeschbach, S., Bakker, M. A., Becker, J. A., Berditchevskaia, A., Berger, J., Brinkmann, L. U. L., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01959-9 · Cited by 158
“For example, multi-agent systems in which multiple LLMs are allowed to converse with one another display enhanced mathematical and strategic reasoning capabilities 56 and can complete tasks more efficiently than any single LLM 57 .”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“…et al., 2022b, Saunders et al., 2022, Ye et al., 2023, Kim et al., 2023, Akyürek et al., 2023, Madaan et al., 2023, Chen et al., 2023, Gilardi et al., 2023] with research agendas like Recursive Reward Modeling [Leike et al., 2018] and AI Safety via debate [Irving et al., 2018, Du et al., 2023].”
Can Large Language Models Really Improve by Self-critiquing Their Own Plans?
Valmeekam, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.08118 · Cited by 90
Generative AI in Marketing and Principles for Ethical Design and Deployment
Hermann, E., & Puntoni, S. (2024) · Journal of Public Policy & Marketing · doi:10.1177/07439156241309874 · Cited by 121
“…model instances generating, critiquing, and debating their own output to collaboratively reach a more accurate consensus response) that facilitate and foster collaboration between systems, content cross-checking, and eventually collective and complementary intelligences (Du et al. 2023).”
Designing Heterogeneous LLM Agents for Financial Sentiment Analysis
Xing, F. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3688399 · Cited by 88
Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?
Chen, Y., Arkin, J., Zhang, Y., Roy, N., & Fan, C. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610676 · Cited by 152
“Recent work shows that discussion among multiple LLM agents can improve factuality and accuracy[31], [32].”
BackgroundPan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86
Security and Privacy on Generative Data in AIGC: A Survey
Wang, T., Zhang, Y., Qi, S., Zhao, R., Xia, Z., & Weng, J. (2024) · ACM Computing Surveys · doi:10.1145/3703626 · Cited by 128
“[123] prompted multiple language models to debate their viewpoints and reasoning processes over multiple rounds, and finally come up with a unitive”
BackgroundEditable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
Wei, Y., Wang, Z., Lu, Y., Xu, C., Liu, C., Zhao, H., Chen, S., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01428 · Cited by 101
“Based on LLM, many works [3, 23, 29, 65, 76] improve the problem-solving abilities by integrating communication among multiple agents.”
BackgroundCognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72
“…multiple language agents for social simulation (Park et al., 2023; Jinxin et al., 2023; Gao et al., 2023), debate (Chan et al., 2023; Liang et al., 2023b; Du et al., 2023), improved safety (Irving et al., 2018), or collabrative task solving (Qian et al., 2023; Wu et al., 2023; Hong et al., 2023).”
BackgroundBuilding machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 125
Self-citationLarge Language Model based Multi-Agents: A Survey of Progress and Challenges
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X. (2024) · arXiv · doi:10.48550/arxiv.2402.01680 · Cited by 1,713
On scalable oversight with weak LLMs judging strong LLMs
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04622 · Cited by 66
Stance Detection with Collaborative Role-Infused LLM-Based Agents
Lan, X., Gao, C., Jin, D., & Li, Y. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31360 · Cited by 69
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61
Generative Ai in EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity
Novelli, C., Casolari, F., Hacker, P., Spedicato, G., & Floridi, L. (2024) · doi:10.2139/ssrn.4821952 · Cited by 57
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 725
“Unlike previous work like Du et al. (2023), we do not explicitly ask the debater agents to reach a consensus at the end of the debate.”
BackgroundResultInfluentialWu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58
D-Bot: Database Diagnosis System using Large Language Models
Zhou, X., Li, G., Sun, Z., Liu, Z., Chen, W., Wu, J., Liu, J., Feng, R., & Zeng, G. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3675034.3675043 · Cited by 108
Liu, Y., Lo, S. K., Lu, Q., Zhu, L., Zhao, D., Xu, X., Harrer, S., & Whittle, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112278 · Cited by 51
Franceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 54
“Using AI itself to provide feedback is also an option; notably, AI-based feedback is also used outside the RL paradigm, e.g., to provide verbal feedback to be appended to prompts (Shinn et al., 2023) or to collaborate with other LLMs at inference time (Dong et al., 2023; Du et al., 2023).”
BackgroundMethodCan ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 127
“Prior work shows that LLMs can improve their responses via feedback (Shinn et al., 2023; Paul et al., 2023; Madaan et al., 2023; Ganguli et al., 2023; Ma et al., 2023; Chen et al., 2023b; Peng et al., 2023; Kim et al., 2023; Du et al., 2023; Liang et al., 2023; Chen et al., 2023a; Pan et al., 2023).”
BackgroundChain of Agents: Large Language Models Collaborating on Long-Context Tasks
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., & Arik, S. Ö. (2024) · arXiv · doi:10.48550/arxiv.2406.02818 · Cited by 47
“Others advance problem solving, focusing on reasoning of short text via multi-agent debating [18, 80, 10, 66] and discussing [11, 58] for different tasks in reasoning [18, 66], mechanics problems [49], paper review [84], knowledge graph construction [86], and code intelligence [70, 24].”
Romero, O. J., Zimmerman, J., Steinfeld, A. M., & Tomasic, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27706 · Cited by 56
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
Chen, J., Prasad, A., Saha, S., Stengel-Eskin, E., & Bansal, M. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1660 · Cited by 38
Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4
Pelrine, K., Imouza, A., Thibault, C., Reksoprodjo, M., Gupta, C., Christoph, J. N., Godbout, J., & Rabbany, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.395 · Cited by 83
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
“Transitioning to another facet of self-correction, we investigate the potential of multi-agent debate (Du et al., 2023; Liang et al., 2023) as a means to improve reasoning.”
BackgroundInfluentialBaek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.342 · Cited by 46
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23
Mitigating reasoning hallucination through Multi-agent Collaborative Filtering
Shi, J., Zhao, J., Wu, X., Xu, R., Jiang, Y.-H., & He, L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125723 · Cited by 34
Large Language Models for Telecom: The Next Big Thing?Preprint
Bariah, L., Zhao, Q., Zou, H., Tian, Y. T., Bader, F., & Debbah, M. A. (2023) · doi:10.36227/techrxiv.23536440.v1 · Cited by 36
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
“Researchers have found taking multiple LLM instances to debate for a fixed number of rounds can boost their factuality and reasoning capabilities [41, 77, 140].”
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 386
“Du et al. (2023) establishes a mechanism for symmetric discussions among agents.”
BackgroundCroxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M., Wills, G., Chen, G., Dligach, D., et al. (2025) · doi:10.1101/2025.04.22.25326219 · Cited by 25
Wu, M., Xu, J., Yuan, Y., Haffari, G., Wan, L., Luo, W., & Zhang, K. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.25 · Cited by 91
R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’
Zhang, H., Diao, S., Lin, Y., Fung, Y., Lian, Q., Wang, X., Chen, Y., Ji, H., & Zhang, T. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.394 · Cited by 28
How AI agents will change cancer research and oncology
Lee, Y., Ferber, D., Rood, J., Regev, A., & Kather, J. N. (2024) · Nature Cancer · doi:10.1038/s43018-024-00861-7 · Cited by 23
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 22
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2023) · arXiv · doi:10.48550/arxiv.2309.03883 · Cited by 29
“Recent strategies involve inference-time self-consistency checks (Manakul et al., 2023), multi-agent debating (Du et al., 2023; Liang et al., 2023), and inference-time intervention using human labels (Li et al., 2023).”
BackgroundLi, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024) · arXiv · doi:10.48550/arxiv.2402.05120 · Cited by 253
Meissner, R., Pögelt, A., Ihsberner, K., Grüttmüller, M., Tornack, S., Thor, A., Pengel, N., Wollersheim, H.-W., & Hardt, W. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1427502 · Cited by 29
“…chain, resulting in a fine-tuned Zero Shot Chain-of-Thought prompting approach (Wei et al., 2022a; Kojima et al., 2022), which incorporates a prompt template and injection of specially prepared knowledge (Martino et al., 2023), augmented by a simplified Multiple-Agent approach (Du et al., 2024).”
Improving Multi-Agent Debate with Sparse Communication Topology
Li, Y., Du, Y., Zhang, J., Hou, L., Grabowski, P., Li, Y., & Ie, E. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.427 · Cited by 20
Self-citationBellini-Leite, S. C. (2023) · Adaptive Behavior · doi:10.1177/10597123231206604 · Cited by 25
“Self-Consistency differs from society of mind approaches (Du et al., 2023; Zhuge et al., 2023), that have been recently proposed, which involves the utilization of multiple language model instances to collaboratively generate a uni fi ed answer to a given query.”
Lu, Q., Zhu, L., Xu, X., Xing, Z., Harrer, S., & Whittle, J. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00028 · Cited by 38
“Debate-based cooperation integrates concepts from game theory, where one agent can receive feedback from other agents [52], [53] and adjust the thoughts and behaviours until consensus is reached [54], [55].”
BackgroundAgentCoord: Visually exploring coordination strategy for LLM-based multi-agent collaboration
Pan, B., Lu, J., Wang, K., Li, Z., Wen, Z., Feng, Y., Zhu, M., & Chen, W. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104338 · Cited by 27
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
“Our empirical analysis reveals 41% to 86.7% failure rate on 7 state-of-the-art (SOTA) open-source MAS detailed in Figure 5 (Appendix C).”
InfluentialThe bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15
On the Calibration of Large Language Models and Alignment
Zhu, C., Xu, B., Wang, Q., Zhang, Y., & Mao, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.654 · Cited by 50
Voting or Consensus? Decision-Making in Multi-Agent Debate
Kaesberg, L. B., Becker, J., Wahle, J. P., Ruas, T., & Gipp, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.606 · Cited by 55
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
Amayuelas, A., Yang, X., Antoniades, A., Hua, W., Pan, L., & Wang, W. Y. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.407 · Cited by 15
AI agents in chemical research: GVIM – an intelligent research assistant system
Ma, K. (2025) · Digital Discovery · doi:10.1039/d4dd00398e · Cited by 16
Lee, P., Son, M., & Jia, Z. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10067-y · Cited by 19
“Such interfaces have been successfully applied across a range of domains, including software development (Qian et al., 2023), games (Park et al., 2023), robotics (Zhang et al., 2023), economics (Li et al., 2023), political science (Xiao et al., 2023), and public health (Williams et al., 2023).”
Sustainable Digitalization of Business with Multi-Agent RAG and LLM
Arslan, M., Munawar, S., & Cruz, C. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.337 · Cited by 17
“Multi-Agent Debate is explored [31, 32] as an effective method to stimulate divergent thinking and enhance factuality and reasoning in LLMs. MetaGPT [33] introduces a specialized LLM application for automatic software development.”
Debating with More Persuasive LLMs Leads to More Truthful Answers
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., & Perez, E. (2024) · arXiv · doi:10.48550/arxiv.2402.06782 · Cited by 13
Schnepf, J., Engin, T., Anderer, S., & Scheuermann, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_2 · Cited by 12
Zero-shot Knowledge Graph Question Generation via Multi-agent LLMs and Small Models Synthesis
Zhao, R., Tang, J., Zeng, W., Chen, Z., & Zhao, X. H. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679805 · Cited by 14
Stock Market Forecasting: From Traditional Predictive Models to Large Language Models
Darwish, M., Hassanien, E. E., & Eissa, A. H. B. (2025) · Computational Economics · doi:10.1007/s10614-025-11024-w · Cited by 28
“…and 5 , some research studies show that performance can be enhanced through fine-tun - ning LLMs with financial tailored information using LoRA (Zhang & Yang, 2023) or hybrid tuning methodology (Raffel et al., 2020 ) or building specialized financial LLMs (Li et al., 2023 ; Le Scao et al., 2022 ).”
InfluentialEvil Geniuses: Delving into the Safety of LLM-based Agents
Tian, Y., Yang, X., Zhang, J., Dong, Y., & Su, H. (2023) · arXiv · doi:10.48550/arxiv.2311.11855 · Cited by 10
Generative AI Agents for Knowledge Work Augmentation in Finance
Ganesh, S., Ardon, L., Borrajo, D., Garg, D., Madhushani, U., Narayanan, A. L., Canonaco, G., & Veloso, M. (2024) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-022823-033813 · Cited by 13
Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labels
Nguyen, V., & Pham, C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825313 · Cited by 18
“[22] organized multiple LLMs in a debate setting, where each LLM independently generates an initial response, and then communicates to reach a consensus on a final answer.”
A brain-inspired agentic architecture to improve planning with LLMs
Webb, T. W., Mondal, S. S., & Momennejad, I. (2025) · Nature Communications · doi:10.1038/s41467-025-63804-5 · Cited by 36
“…a version of GPT-4 that was also given access to detailed explanations of those solutions, sometimes referred to as chain-of-thought 36 (GPT-4 CoT); Tree-of-Thought (ToT) 37 , a method that combines LLMs with tree search, similar to the search performed by MAP; and Multi-Agent Debate (MAD) 38 .”
MARG: Multi-Agent Review Generation for Scientific Papers
D'Arcy, M., Hope, T., Birnbaum, L., & Downey, D. (2024) · arXiv · doi:10.48550/arxiv.2401.04259 · Cited by 9
Xue, J., Yu, J., Mao, Q., & Gu, X. (2025) · EngMedicine · doi:10.1016/j.engmed.2025.100086 · Cited by 10
Auto-scaling LLM-based multi-agent systems through dynamic integration of agents
Perera, R., Basnayake, A., & Wickramasinghe, M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1638227 · Cited by 10
“Multi-agent debate is an effective way to improve reasoning, adaptability, and factual accuracy in language models (Du et al., 2023; Talebirad and Nadiri, 2023).”
Automated Design of Agentic Systems
Hu, S., Lu, C., & Clune, J. (2024) · arXiv · doi:10.48550/arxiv.2408.08435 · Cited by 10
“…(Madaan et al., 2024; Shinn et al., 2023), which allows iterative self-reflection to correct mistakes made in previous attempts; (4) LLM-Debate (Du et al., 2023), which enables different LLMs to debate with each other, leveraging diverse perspectives to find better answers; (5)…”
InfluentialLarge Language Models for Telecom: The Next Big Thing?Preprint
Bariah, L., Zhao, Q., Zou, H., Tian, Y. T., Bader, F., & Debbah, M. A. (2023) · doi:10.36227/techrxiv.23536440 · Cited by 7
EcoAssistant: Using LLM Assistant More Affordably and Accurately
Zhang, J., Krishna, R., Awadallah, A. H., & Wang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.03046 · Cited by 8
“, 2023) or to improve factuality and reasoning (Du et al., 2023).”
BackgroundScaling Large Language Model-based Multi-Agent Collaboration
Qian, C., Xie, Z., Wang, Y., Liu, W., Zhu, K., Xia, H., Dang, Y., Du, Z., Chen, W., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07155 · Cited by 8
“In contrast, agent coordination, based on the implicit knowledge of pretrained LLMs, leverages the understanding and refinement of textual information through linguistic interactions, often circumventing the extensive scaling needed by neuronal coordination.”
TradingAgents: Multi-Agents LLM Financial Trading Framework
Xiao, Y., Sun, E., Luo, D., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.20138 · Cited by 6
“These concise, focused discussions have been shown to promote deeper reasoning and integrate diverse perspectives, enabling more balanced decisions in complex, long-horizon scenarios—a method particularly relevant to the intricate environment of trading (Du et al., 2023).”
Navigating Bias: Using LLMs to Analyze Discrimination in Entrepreneurial Game Dialogues
Marsden, N., Rothe‐Wulf, A., Tang, Y., Herling, C., & Reichert, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92578-8_14 · Cited by 6
An LLM-based multi-agent system for remote sensing analysis
Sun, Z., Zhou, Y., & Yang, J. (2026) · Big Earth Data · doi:10.1080/20964471.2025.2600178 · Cited by 10
“Since multi-agent systems have been shown to improve scalability and reasoning effectiveness (Du et al., 2023; Wu et al., 2023), we adopt a multi-agent structure, setting it apart from existing tools that rely on single-agent architectures.”
InfluentialPrompt Engineering Through the Lens of Optimal Control
Luo, Y., Tang, Y., Shen, C., Zhou, Z., & Dong, B. (2023) · Journal of Machine Learning · doi:10.4208/jml.231023 · Cited by 28
“Tipping the Balance”: Human Intervention in Large Language Model Multi‐Agent Debate
Triem, H., & Ding, Y. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1034 · Cited by 9
“Recently, multiagent work has centered around two broad structures: “collaboration” and “debate,” both of which allow for improved complexity in discussion (Du et al., 2023; H. Li et al., 2023; K. Xiong et al., 2023; Liang et al., 2023).”
Collaborative garment design through group chatting with generative industrial large models
Harish, A. R., Yuan, Z., Li, M., Yang, H., & Huang, G. Q. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103366 · Cited by 6
Learning from Mistakes via Cooperative Study Assistant for Large Language Models
Wang, D., & Li, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.659 · Cited by 17
“Du et al. (2023) uses responses from multiple agents and the debate between these agents as the feedback.”
BackgroundLarge language models for biomolecular analysis: From methods to applications
Feng, R., Zhang, C., & Zhang, Y. (2024) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2024.117540 · Cited by 7
Social Learning in Neural Agent-Based Models
Douven, I. (2024) · Philosophy of Science · doi:10.1017/psa.2024.33 · Cited by 6
“Recent work has shown how LLMs can be made to communicate in that one LLM ’ s output serves as the prompt for one or more other LLMs, and so on, recursively (Du et al. 2023).”
A Self-Evolving Framework for Multi-Agent Medical Consultation Based on Large Language Models
Chen, K., Qi, J., Huo, J., Tian, P., Meng, F., Yang, X., & Gao, Y. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10889517 · Cited by 6
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 24
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A. (2023) · arXiv · doi:10.48550/arxiv.2311.17371 · Cited by 73
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1354 · Cited by 105
Metadata
- Authors
- Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.14325
- OpenAlex
- W4378509427
- Cited by
- 1,202
- References
- 48
- Citations by year
- 2026: 547 · 2025: 340 · 2024: 258 · 2023: 56
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, factuality, hallucination mitigation, mathematical reasoning, strategic reasoning, Society of Mind, self-consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
