The DOI matches its DataCite record; 47 of the 48 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Open accessAbstract
Large language models (LLMs) have demonstrated remarkable capabilities in language generation, understanding, and few-shot learning in recent years. An extensive body of work has explored how their performance may be further improved through the tools of prompting, ranging from verification, self-consistency, or intermediate scratchpads. In this paper, we present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate. Overall, our findings suggest that such "society of minds" approach has the potential to significantly advance the capabilities of LLMs and pave the way for further breakthroughs in language generation and understanding.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Multiagent debate, where multiple LLM instances propose answers and critique each other over rounds, improves reasoning and factual accuracy over single-agent, reflection, and majority-vote baselines across six tasks. The approach uses black-box models and identical prompts, though it is more computationally expensive and can converge confidently on incorrect answers.
Key findings
- Debate improved reasoning: arithmetic accuracy was 81.8 ± 2.3 vs 67.0 ± 4.7 for single agent, 72.1 ± 4.5 for reflection, and 69.0 ± 4.6 for majority voting; GSM8K was 85.0 ± 3.5 vs 77.0 ± 4.2, 75.0 ± 4.3, and 81.0 ± 3.9; chess pawn score was 122.9 ± 7.6 vs 91.4 ± 10.6, 102.1 ± 11.9, and 102.2 ± 6.2.
- Debate improved factual validity: biographies accuracy was 73.8 ± 2.3 vs 66.0 ± 2.2 for single agent and 68.3 ± 2.9 for reflection; MMLU was 71.1 ± 4.6 vs 63.9 ± 4.8 and 57.7 ± 5.0; chess move validity was 45.2 ± 2.9 vs 29.3 ± 2.6 and 38.8 ± 2.9.
- On 20 GSM8K problems debated between ChatGPT and Bard, Bard solved 11, ChatGPT solved 14, and joint multi-agent debate solved 17.
- Performance increased with more agents and more debate rounds; additional rounds above four gave similar final performance to 4 rounds. Longer debate prompts led to slower convergence but better final consensus, and different persona prompts improved MMLU from 71.1 to 74.2.
Methods
- Multiple language model instances independently answer a query, then each reads and critiques the other agents’ responses and updates its own answer over multiple rounds; the same procedure and prompts are used across tasks, requiring only black-box model access.
- Evaluated on six tasks: arithmetic (100 generated tasks), GSM8K (100 problems), chess move prediction (300 games), biographies (524 well-known computer scientists), MMLU (100 questions), and chess move validity (100 tasks), using gpt-3.5-turbo-0301; main debate runs used 3 agents and 2 rounds.
- Compared against single-agent, reflection, and majority-voting baselines; reported accuracy, Stockfish pawn score, and factual validity, with ablations on number of agents, rounds, prompt length, summarization, and different models/personas.
Limitations
- Stated: multiagent debate is more computationally expensive because it requires multiple generations and debate rounds; longer debates sometimes exceed models’ processing, with models focusing mainly on recent generations; debates converge but not necessarily to correct answers, and models can confidently affirm incorrect answers because they do not correctly express uncertainty.
- Stated: the biography evaluation metric may miss incorrect generated facts not captured in the ground-truth bullets.
- Evident from the text: main evaluations often use 100 examples per task and 3 agents with 2 rounds due to computational expense, and the cross-model comparison uses only 20 GSM8K problems.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
44 Own result · 11 No basis given · 10 Own interpretation · 4 Own method · 2 Sources not found · 1 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2: Multiagent Debate Improves Factual Accuracy Multi-agent debate improves the factual accuracy.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Model Biographies MMLU Chess Move Validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate results are also less likely to include false facts that models are internally uncertain of.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we found that additional debate rounds above four led to a similar final performance to 4 rounds of debate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On arithmetic, performance monotonically increases with the increased number of agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In comparison to other prompting techniques, our multiagent debate procedure is more computationally expensive, as it requires both multiple language generations, and an underlying debate procedure.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After debating and examining the responses of other model instances, we find that the population almost always converges on a single and more accurate common answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, directly asking each agent about their confidence [10] of the answer led to high confidence assessments on each answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
What the sources say
- Not matchedcites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, these facts often inconsistent across different language model instances.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preliminary testing, we found that existing language models had a tendency to hallucinate many facts on this task.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 3.2 Extracting Factual Information from Multiagent Debate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we found that on facts that the language model was confident in (i.e.many instances of the same model all gave the same answer), it was very difficult to convince an agent to change their opinion, suggesting that "ease of persuasion" may be a method to assess factual confidence.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that different language agents tended to give different answers when the underlying language model was uncertain about the question.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, we also see that reflection, where a language model is asked to critique its early generation, generally gives a modest boost in performance.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Quantitative Results.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, however, we find that language models are able to converge on a single shared answer after multiple rounds of debate (Figure 4).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that we could control the duration of debates by how changing how much a language model trusts its own outputs over those generated by other models through different prompts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that approaches based on reflection led to poor performance in the factuality setting.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lastly, we find that debate does not just act to amplify one correct answer in a model quorum -we find many cases where all the models initially make incorrect predictions, but then arrive at the correct answer as debate progresses (Figure 4,11).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Multiagent debate, which may be seen as a combination of both reflection and multiagent generation, gives a substantial boost in reasoning across each of the tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Quantitative Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we find cases in which all models initially give an incorrect response, yet the result of debate still obtains the correct answer as agents critique each others' reasoning.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that on the arithmetic task, the performance also monotonically increases with debate length.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Number of Agents.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“We find that our debate approach outperforms single model baselines such as zero-shot chain of thought [11] and reflection [26,18] on a variety of six reasoning, factuality, and question-answering tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that contemporary language models have an especially high tendency to hallucinate factually incorrect biographies, often misrepresenting the relevant institutions and dates.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite answers being incorrect, language models would confidently affirm that their answer is correct and consistent with all other agent responses.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find this improves the performance of debate, suggesting that summarization is another tool that can further improve multiagent debate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that improved performance on MMLU from 71.1 to 74.2 with different agents, suggesting further gains can be obtained with different initialization prompts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when these different language agents were asked to communicate with each other, each agent would quickly change their opinion to a consensus answer which was more accurate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each task, we observe that utilizing multiple different agents to generate solutions improves performance over using a single language model agent to generate a solution.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Quantitative Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given an initial query, we find that individual model instances propose a diverse range of answers despite being the same model class”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performance of debate increases with use of Chain of Thought prompting.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This operation improves performance.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Multi-agent debate improves the reasoning abilities of language models.”
- Also stated in: § Agent 1: · plain
“Multiagent Debate Improves Reasoning”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Agent 1: · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For facts that a language model is uncertain about, different language agents generate different facts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we found that generated facts could contain incorrect information that was not captured in the ground truth bullet and thus could not be validated through this metric.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Biographies.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Multiagent Debate Improves Reasoning and Factual Accuracy.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The use of debate prompts that encourage agents to adapt more to the opinions of other agents improves consensus.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Consensus Between Agents.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By asking models to come to a consensus across their answers, such inconsistent facts may be either removed or corrected.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that summarization is another tool that can further improve multiagent debate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that "ease of persuasion" may be a method to assess factual confidence.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LMs do not correctly express their uncertainty when generating responses”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This allows our method to be used with common public models serving interfaces.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The resulting quorum of models can hold and maintain multiple chains of reasoning and possible answers simultaneously before proposing the final answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This process induces models to construct answers that are consistent with both their internal critic as well as sensible in light of the responses of other agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 9
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this set, we find that multi-agent debate improves the performance of both agents, with Bard solving 11 problems, chatGPT solving 14 problems, and joint multi-agent debate solving 17 problems.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using both multiple model agents and multiple rounds of debate are important to achieve the best performance.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because as the debate progresses, individual model instances tend to disagree on uncertain facts and omit them from the answer (Figure 7).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both settings, multiagent generation is beneficial.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the approach is simple and effective across a wide set of different reasoning and validity language modeling tasks.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 5 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that multiagent debate improved and settled on bullets that were more consistent across agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although we found further gains with both more agents and rounds of debate (Figure 10).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we found that while debates typically converged into single final answers, these answers were not necessarily correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While both agents initially provide incorrect answers to the problem, chatGPT is able to utilize the incorrect response given by Bard to generate the final correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > ✗
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Overall, we observed that language model agents were relatively "agreeable"”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that prompts that encouraged models to be more "stubborn' based on their own solutions led to longer debates and better final solutions.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.2 Consensus in Debates
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, we observed that as debates became longer in duration, current language models sometimes struggled to fully process the entire debate input, and typically only focused on the most recent generations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Limitations.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that debates using longer prompts lead to slower convergence to correct answers, but also lead to a better final consensus on the correct answer.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Effect of Debate Length on Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, debate gives the best performance in this setting also, and significantly outperforms each baseline.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
- Rests on an assumption: Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When there are many agents in a debate, responses from other agents may be first summarized and then given as context, reducing context length.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Debate causes agents to converge to one fact that is more accurate, but not necessarily always factually correct.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we found that generated biographies after debate to be more accurate, many facts remain incorrect.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we found that debate improved the performance of final generated answers, though sometimes answers would converge to the incorrect value.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Additional Qualitative Visualizations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using a short debate prompt induces faster consensus between agents”
- Also stated in: § Agent 1: · plain
“Using a short debate prompt induces faster consensus between agents”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > A Appendix
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Agent 1: · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Model | Arithmetic (%) ↑ Grade School Math (%) ↑ Chess (∆PS) ↑ | | | | --- | --- | --- | --- | | Single Agent | 67.0 ± 4.7 | 77.0 ± 4.2 | 91.4 ± 10.6 | | Single Agent (Reflection) | 72.1 ± 4.5 | 75.0 ± 4.3 | 102.1 ± 11.9 | | Multi-Agent (Majority) | 69.0 ± 4.6 | 81.0 ± 3.9 | 102.2 ± 6.2 | | Multi-Agent (Debate) | 81.8 ± 2.3 | 85.0 ± 3.5 | 122.9 ± 7.6 |”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The method is also orthogonal to other model generation improvements such as retrieval or prompt engineering (in fact, we combine our debate method with zero-shot chain of thought).”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to computational expense, we evaluate our approach across benchmarks mainly using three agents with two rounds of debates”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As the focus of our experiments is to verify the effectiveness of multiagent agent debate, we run both baselines and our approach, using the identical starting prompt and language model across all evaluations.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the purpose of our debate isn't just to amplify a correct answer -all models can initially be wrong but arrive at the correct answer through the debate process.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Agent 1:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting further gains can be obtained with different initialization prompts.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.”
- Also stated in: § 1 Introduction · plain
“We use the same methodology and prompt templates for all our tasks and require only black-box access to language model generations -no model-internal information such as likelihoods or gradients is needed.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments we use the same prompts for all agents.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Using Different Initialization Prompts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The constructed ground truth bullet point biographies of 524 well-known computer scientists are accurate.
“We constructed ground truth bullet point biographies of 524 well-known computer scientists.”
The paper relies on this without support.
The chatGPT-based language model used in the experiments is representative of contemporary language models.
“We use chatGPT-based language model [21] in all our experiments except those in Figure 11 where we compare multiple language models.”
The paper relies on this without support.
Evaluating factual accuracy by checking agreement between generated bullets and ground truth bullet points is a valid measure of factual accuracy.
“We then asked language models to generate bullet point biographies for each person, and evaluated the accuracy at which each ground truth bullet point agreed with generated bullets.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this method, the capability of each agent is enhanced by understanding and incorporating the other agents' opinions.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [91]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“consisting of six different two-digit numbers”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Tasks.
“For example: What is the result of 12+15*21+0-3*27?”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“0.5 × 3 × 4 = 64”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > 2.1 Multiagent Language Generation
Reported 64 · recomputed 6
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Abstract
“uses identical procedure and prompts for all tasks we investigate.”
§ Improving Factuality and Reasoning in Language Models through Multiagent Debate > Arithmetic
“## Arithmetic Starting What is the result of {}+{}*{}+{}-{}*{}?Make sure to state your answer at the end of the response.”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“To improve reasoning, approaches have relied on prompting techniques such as scratchpads [20], verification [3], chain-of-thought demonstrations [30, 11, 25], and intermediate self-reflection [26, 18] and finetuning [13, 24, 31].”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Overall, we observed that language model agents were relatively "agreeable", perhaps as a result of instruction tuning or reinforcement learning based on human feedback (Ouyang et al., 2022).”
Minsky, M. (1987) · Personalist Forum · doi:10.5840/persforum19873112 · Cited by 2,415 · cited 2×
“Instead, we propose a complementary approach inspired by The Society of Mind (Minsky, 1988) and multiagent settings, where multiple language model instances (or agents) individually propose and jointly debate their responses and reasoning processes to arrive at a common answer.”
InfluentialSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 2×
“We find that our debate approach outperforms single model baselines such as zero-shot chain of thought (Kojima et al., 2022) and reflection (Shinn et al., 2023; Madaan et al., 2023) on a variety of six reasoning, factuality, and question-answering tasks.”
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3×
“In Figure 6, we illustrate the performance of multi-agent debate with and without zero-shot chain-ofthought prompting [11] on GSM8K.”
BackgroundMethodLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Research works printed but not linked: 1
- Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 9
Cited by
Sorted by citations
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 1,062
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“For example, Du et al. [92] introduced the concept of debate, endowing agents with responses from fellow peers.”
InfluentialChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
“Even if the possibility of LLM hallucination is deemed to be likely in certain assessments, recent studies demonstrate that with careful strategies and prompting techniques, hallucination can to a large degree be mitigated, though not entirely eliminated [22,50–53].”
Metadata
- Authors
- Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.14325
- OpenAlex
- W4378509427
- Cited by
- 1,202
- References
- 48
- Citations by year
- 2026: 547 · 2025: 340 · 2024: 258 · 2023: 56
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, factuality, hallucination mitigation, mathematical reasoning, strategic reasoning, Society of Mind, self-consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
