The DOI matches its DataCite record; 47 of the 48 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Open accessAbstract
Large language models (LLMs) have demonstrated remarkable capabilities in language generation, understanding, and few-shot learning in recent years. An extensive body of work has explored how their performance may be further improved through the tools of prompting, ranging from verification, self-consistency, or intermediate scratchpads. In this paper, we present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate. Overall, our findings suggest that such "society of minds" approach has the potential to significantly advance the capabilities of LLMs and pave the way for further breakthroughs in language generation and understanding.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Multiagent debate, where multiple LLM instances propose answers and critique each other over rounds, improves reasoning and factual accuracy over single-agent, reflection, and majority-vote baselines across six tasks. The approach uses black-box models and identical prompts, though it is more computationally expensive and can converge confidently on incorrect answers.
Key findings
- Debate improved reasoning: arithmetic accuracy was 81.8 ± 2.3 vs 67.0 ± 4.7 for single agent, 72.1 ± 4.5 for reflection, and 69.0 ± 4.6 for majority voting; GSM8K was 85.0 ± 3.5 vs 77.0 ± 4.2, 75.0 ± 4.3, and 81.0 ± 3.9; chess pawn score was 122.9 ± 7.6 vs 91.4 ± 10.6, 102.1 ± 11.9, and 102.2 ± 6.2.
- Debate improved factual validity: biographies accuracy was 73.8 ± 2.3 vs 66.0 ± 2.2 for single agent and 68.3 ± 2.9 for reflection; MMLU was 71.1 ± 4.6 vs 63.9 ± 4.8 and 57.7 ± 5.0; chess move validity was 45.2 ± 2.9 vs 29.3 ± 2.6 and 38.8 ± 2.9.
- On 20 GSM8K problems debated between ChatGPT and Bard, Bard solved 11, ChatGPT solved 14, and joint multi-agent debate solved 17.
- Performance increased with more agents and more debate rounds; additional rounds above four gave similar final performance to 4 rounds. Longer debate prompts led to slower convergence but better final consensus, and different persona prompts improved MMLU from 71.1 to 74.2.
Methods
- Multiple language model instances independently answer a query, then each reads and critiques the other agents’ responses and updates its own answer over multiple rounds; the same procedure and prompts are used across tasks, requiring only black-box model access.
- Evaluated on six tasks: arithmetic (100 generated tasks), GSM8K (100 problems), chess move prediction (300 games), biographies (524 well-known computer scientists), MMLU (100 questions), and chess move validity (100 tasks), using gpt-3.5-turbo-0301; main debate runs used 3 agents and 2 rounds.
- Compared against single-agent, reflection, and majority-voting baselines; reported accuracy, Stockfish pawn score, and factual validity, with ablations on number of agents, rounds, prompt length, summarization, and different models/personas.
Limitations
- Stated: multiagent debate is more computationally expensive because it requires multiple generations and debate rounds; longer debates sometimes exceed models’ processing, with models focusing mainly on recent generations; debates converge but not necessarily to correct answers, and models can confidently affirm incorrect answers because they do not correctly express uncertainty.
- Stated: the biography evaluation metric may miss incorrect generated facts not captured in the ground-truth bullets.
- Evident from the text: main evaluations often use 100 examples per task and 3 agents with 2 rounds due to computational expense, and the cross-model comparison uses only 20 GSM8K problems.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
44 Own result · 11 No basis given · 10 Own interpretation · 4 Own method · 2 Sources not found · 1 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2: Multiagent Debate Improves Factual Accuracy Multi-agent debate improves the factual accuracy.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Model Biographies MMLU Chess Move Validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate results are also less likely to include false facts that models are internally uncertain of.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we found that additional debate rounds above four led to a similar final performance to 4 rounds of debate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On arithmetic, performance monotonically increases with the increased number of agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In comparison to other prompting techniques, our multiagent debate procedure is more computationally expensive, as it requires both multiple language generations, and an underlying debate procedure.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After debating and examining the responses of other model instances, we find that the population almost always converges on a single and more accurate common answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, directly asking each agent about their confidence [10] of the answer led to high confidence assessments on each answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, these facts often inconsistent across different language model instances.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preliminary testing, we found that existing language models had a tendency to hallucinate many facts on this task.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 3.2 Extracting Factual Information from Multiagent Debate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we found that on facts that the language model was confident in (i.e.many instances of the same model all gave the same answer), it was very difficult to convince an agent to change their opinion, suggesting that "ease of persuasion" may be a method to assess factual confidence.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 9
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this set, we find that multi-agent debate improves the performance of both agents, with Bard solving 11 problems, chatGPT solving 14 problems, and joint multi-agent debate solving 17 problems.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using both multiple model agents and multiple rounds of debate are important to achieve the best performance.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because as the debate progresses, individual model instances tend to disagree on uncertain facts and omit them from the answer (Figure 7).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both settings, multiagent generation is beneficial.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the approach is simple and effective across a wide set of different reasoning and validity language modeling tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 5 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that multiagent debate improved and settled on bullets that were more consistent across agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although we found further gains with both more agents and rounds of debate (Figure 10).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we found that while debates typically converged into single final answers, these answers were not necessarily correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While both agents initially provide incorrect answers to the problem, chatGPT is able to utilize the incorrect response given by Bard to generate the final correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Overall, we observed that language model agents were relatively "agreeable"”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that prompts that encouraged models to be more "stubborn' based on their own solutions led to longer debates and better final solutions.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, we observed that as debates became longer in duration, current language models sometimes struggled to fully process the entire debate input, and typically only focused on the most recent generations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that debates using longer prompts lead to slower convergence to correct answers, but also lead to a better final consensus on the correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Effect of Debate Length on Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, debate gives the best performance in this setting also, and significantly outperforms each baseline.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When there are many agents in a debate, responses from other agents may be first summarized and then given as context, reducing context length.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate causes agents to converge to one fact that is more accurate, but not necessarily always factually correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we found that generated biographies after debate to be more accurate, many facts remain incorrect.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that debate improved the performance of final generated answers, though sometimes answers would converge to the incorrect value.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional Qualitative Visualizations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using a short debate prompt induces faster consensus between agents”
- Also stated in: § Agent 1: · plain
“Using a short debate prompt induces faster consensus between agents”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > A Appendix
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Agent 1: · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The method is also orthogonal to other model generation improvements such as retrieval or prompt engineering (in fact, we combine our debate method with zero-shot chain of thought).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to computational expense, we evaluate our approach across benchmarks mainly using three agents with two rounds of debates”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As the focus of our experiments is to verify the effectiveness of multiagent agent debate, we run both baselines and our approach, using the identical starting prompt and language model across all evaluations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the purpose of our debate isn't just to amplify a correct answer -all models can initially be wrong but arrive at the correct answer through the debate process.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting further gains can be obtained with different initialization prompts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.”
- Also stated in: § 1 Introduction · plain
“We use the same methodology and prompt templates for all our tasks and require only black-box access to language model generations -no model-internal information such as likelihoods or gradients is needed.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments we use the same prompts for all agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
“We constructed ground truth bullet point biographies of 524 well-known computer scientists.”
The paper relies on this without support.
The chatGPT-based language model used in the experiments is representative of contemporary language models.
“We use chatGPT-based language model [21] in all our experiments except those in Figure 11 where we compare multiple language models.”
The paper relies on this without support.
Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
“We then asked language models to generate bullet point biographies for each person, and evaluated the accuracy at which each ground truth bullet point agreed with generated bullets.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this method, the capability of each agent is enhanced by understanding and incorporating the other agents' opinions.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“consisting of six different two-digit numbers”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“For example: What is the result of 12+15*21+0-3*27?”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“0.5 × 3 × 4 = 64”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.1 Multiagent Language Generation
Reported 64 · recomputed 6
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
“uses identical procedure and prompts for all tasks we investigate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Arithmetic
“## Arithmetic Starting What is the result of {}+{}*{}+{}-{}*{}?Make sure to state your answer at the end of the response.”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“To improve reasoning, approaches have relied on prompting techniques such as scratchpads [20], verification [3], chain-of-thought demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and finetuning [13, 24, 31].”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Overall, we observed that language model agents were relatively "agreeable", perhaps as a result of instruction tuning or reinforcement learning based on human feedback (Ouyang et al., 2022).”
Minsky, M. (1987) · Personalist Forum · doi:10.5840/persforum19873112 · Cited by 2,415 · cited 2×
“Instead, we propose a complementary approach inspired by The Society of Mind (Minsky, 1988) and multiagent settings, where multiple language model instances (or agents) individually propose and jointly debate their responses and reasoning processes to arrive at a common answer.”
InfluentialSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 2×
“We find that our debate approach outperforms single model baselines such as zero-shot chain of thought (Kojima et al., 2022) and reflection (Shinn et al., 2023; Madaan et al., 2023) on a variety of six reasoning, factuality, and question-answering tasks.”
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3×
“In Figure 6, we illustrate the performance of multi-agent debate with and without zero-shot chain-ofthought prompting [11] on GSM8K.”
BackgroundMethodLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 7×
“State the destination square in the form (X), where X follows the regex [a-h][1-8], for example (e5).”
InfluentialCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 1×
“Our work is also related to existing works that focus on text generation by combining different models (Du et al., 2020; Liu et al., 2022b; Zeng et al., 2022; Alayrac et al., 2022; Du et al., 2023).”
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 2×
“…such as scratchpads (Nye et al., 2021), verification (Cobbe et al., 2021), chain-of-thought demonstrations (Wei et al., 2022; Kojima et al., 2022; Reynolds & McDonell, 2021), and intermediate self-reflection (Shinn et al., 2023; Madaan et al., 2023) and finetuning (Lewkowycz et al., 2022; Rajani…”
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1×
“Prior work also has explored how to take the majority vote across different models [15, 3, 29, 28] while in this work, we use the power of a language model to combine different answers.”
MethodLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1×
“Prior work also has explored how to take the majority vote across different models (Li et al., 2022b; Cobbe et al., 2021; Wang et al., 2022; Thoppilan et al., 2022; Lewkowycz et al., 2022) while in this work, we use rounds of debate between language model to combine answers.”
Self-citationChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 725 · cited 3×
“In contrast, our work and work in (Liang et al., 2023; Chan et al., 2023) focus on using debate at inference time to improve the performance of language models.”
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 3×
“Using the GSM8K dataset (Cobbe et al., 2021), the models must correctly solve grade school mathematical reasoning tasks.”
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574 · cited 3×
“State your final answer in a newline with a 2 letter response following the regex [a-h][1-8].”
BackgroundMethodBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 2×
“To evaluate chess validity, we consider the BIG-Bench Chess-State Tracking Benchmark (Srivastava et al., 2022), where we used the hardest reported task in the benchmark synthetic_short .”
Self-citationExplain Yourself! Leveraging Language Models for Commonsense Reasoning
Rajani, N. F., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1487 · Cited by 560 · cited 1×
“To improve reasoning, approaches have relied on prompting techniques such as scratchpads [20], verification [3], chain-of-thought demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and finetuning [13, 24, 31].”
MethodFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · cited 4×
“Our results indicate that RLHF significantly improves consensus in multiagent debate.”
InfluentialCompositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A. B., & Tenenbaum, J. B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19790-1_26 · Cited by 365
Self-citationSolving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358 · cited 3×
“If the butter costs $4.00 a pound, how much will it cost him to purchase 6 pounds of butter? sider generating responses from multiple instances of model and performing majority voting across responses (Wang et al., 2022; Lewkowycz et al., 2022).”
Influential(2022) · doi:10.52202/068431-0278 · Cited by 358
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.19118 · Cited by 320 · cited 4×
“In Table A6, we find that our approach substantially outperforms Liang et al. (2023), as it more effectively encourages language models to focus on reasoning.”
InfluentialCamel: Communicative agents for" mind" exploration of large language model society
Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2264 · Cited by 314 · cited 2×
“Largely concurrent to the work in this paper, a body of recent work has explored how multiagent interaction can be used with large language models (Zhang et al., 2023; Li et al., 2024; Hong et al., 2023).”
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 1×
“Our approach also improves the performance of a GPT-4 model on the MMLU and MATH dataset (Hendrycks et al., 2021).”
Compositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022) · arXiv · doi:10.48550/arxiv.2206.01714 · Cited by 295 · cited 1×
“Our work is also related to existing works that focus on text generation by combining different models [4, 17, 32, 1, 5].”
BackgroundSelf-citationMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 3×
“We utilize the existing MMLU dataset [8] to benchmark the accuracy of responses.”
BackgroundMethodReduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCPreprint
Du, Y., Durkan, C., Strudel, R., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2302.11552 · Cited by 274 · cited 1×
“Our work is also related to existing works that focus on text generation by combining different models (Du et al., 2020; Liu et al., 2022b; Zeng et al., 2022; Alayrac et al., 2022; Du et al., 2023).”
Self-citationMetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214 · cited 2×
“Largely concurrent to the work in this paper, a body of recent work has explored how multiagent interaction can be used with large language models (Zhang et al., 2023; Li et al., 2024; Hong et al., 2023).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205 · cited 2×
“Most similar to our work, (Li et al., 2022a; Zeng et al., 2022) pro-pose to combine multiple different large pretrained models together for multimodal reasoning.”
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · cited 2×
“…have relied on training techniques such as RLHF (Ziegler et al., 2019; Liu et al., 2022a; Christiano et al., 2017), pruning truthful datasets (Lee et al., 2022), external knowledge retrieval (Guu et al., 2020) and training-free methods based off likelihood estimation (Kadavath et al., 2022).”
STaR: Bootstrapping Reasoning With Reasoning
Zelikman, E., Wu, Y., Mu, J., & Goodman, N. D. (2022) · arXiv · doi:10.48550/arxiv.2203.14465 · Cited by 182 · cited 1×
“To improve reasoning, approaches have relied on prompting techniques such as scratchpads [20], verification [3], chain-of-thought demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and finetuning [13, 24, 31].”
Method(2022) · doi:10.52202/068431-1126 · Cited by 182
Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?
Chen, Y., Arkin, J., Zhang, Y., Roy, N., & Fan, C. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610676 · Cited by 152 · cited 2×
“Largely concurrent to the work in this paper, a body of recent work has explored how multiagent interaction can be used with large language models (Zhang et al., 2023; Li et al., 2024; Hong et al., 2023).”
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 1×
“To improve reasoning, approaches have relied on techniques such as scratchpads (Nye et al., 2021), verification (Cobbe et al., 2021), chain-of-thought demonstrations (Wei et al., 2022; Kojima et al., 2022; Reynolds & McDonell, 2021), and intermediate self-reflection (Shinn et al., 2023; Madaan et…”
Irving, G., Christiano, P., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1805.00899 · Cited by 71 · cited 2×
“Most similar to our work, (Irving et al., 2018) also proposes a debate procedure to verify the accuracy and safety of powerful AI agents.”
Factuality Enhanced Language Models for Open-Ended Text Generation
Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M., & Catanzaro, B. (2022) · arXiv · doi:10.48550/arxiv.2206.04624 · Cited by 68 · cited 1×
“To improve factuality, approaches have relied on training techniques such as RLHF [33, 16, 2], pruning truthful datasets [12], external knowledge retrieval [7] and training-free methods based off likelihood estimation [10].”
Method(2022) · doi:10.52202/068431-2506 · Cited by 68
Building Cooperative Embodied Agents Modularly with Large Language Models
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J. B., Shu, T., & Gan, C. (2023) · arXiv · doi:10.48550/arxiv.2307.02485 · Cited by 52
Self-citationInstruction-Following Agents with Multimodal TransformerPreprint
Líu, H., Lee, L., Lee, K., & Abbeel, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.13431 · Cited by 42 · cited 4×
“Our results indicate that RLHF significantly improves consensus in multiagent debate.”
InfluentialComposing Ensembles of Pre-trained Models via Iterative ConsensusPreprint
Li, S., Du, Y., Tenenbaum, J. B., Torralba, A. B., & Mordatch, I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.11522 · Cited by 26 · cited 1×
“Most similar to our work, [14, 32] propose to combine multiple different large pretrained models together for multimodal reasoning.”
BackgroundSelf-citationTwo-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension QuestionsPreprint
Parrish, A., Trivedi, H. K., Nangia, N., Padmakumar, V., Phang, J., Saimbhi, A. S., & Bowman, S. R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10860 · Cited by 20 · cited 3×
“In Parrish et al. (2022) and Michael et al. (2023), debate between two unreliable human experts is used to facilitate a less-informed human judge to answer questions about a passage.”
Debating with More Persuasive LLMs Leads to More Truthful Answers
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., & Perez, E. (2024) · arXiv · doi:10.48550/arxiv.2402.06782 · Cited by 13 · cited 1×
“Most similar to our work, recent works have explored how debate can be applied to language models (Parrish et al., 2022; Michael et al., 2023; Khan et al., 2024).”
BMJ Publishing Group Ltd, & Royal College of Psychiatrists and British Psychological Society (2008) · Evidence-Based Mental Health · doi:10.1136/ebmh.11.4.102 · Cited by 3 · cited 2×
“Prior work also has explored how to take the majority vote across different models [15, 3, 29, 28] while in this work, we use the power of a language model to combine different answers.”
Research works printed but not linked: 1
- Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 9
Cited by
Sorted by citations
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 1,062
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“For example, Du et al. [92] introduced the concept of debate, endowing agents with responses from fellow peers.”
InfluentialChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
“Even if the possibility of LLM hallucination is deemed to be likely in certain assessments, recent studies demonstrate that with careful strategies and prompting techniques, hallucination can to a large degree be mitigated, though not entirely eliminated [22,50–53].”
Metadata
- Authors
- Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.14325
- OpenAlex
- W4378509427
- Cited by
- 1,202
- References
- 48
- Citations by year
- 2026: 547 · 2025: 340 · 2024: 258 · 2023: 56
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, factuality, hallucination mitigation, mathematical reasoning, strategic reasoning, Society of Mind, self-consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
