Voting or Consensus? Decision-Making in Multi-Agent Debate
Open accessPublished version of a preprint
Abstract
Much of the success of multi-agent debates depends on carefully choosing the right parameters.The decision-making protocol stands out as it can highly impact final model answers, depending on how decisions are reached.Systematic comparison of decision protocols is difficult because many studies alter multiple discussion parameters beyond the protocol.So far, it has been largely unknown how decisionmaking influences different tasks.This work systematically evaluates the impact of seven decision protocols (e.g., majority voting, unanimity consensus).We change only one variable at a time -the decision protocol -to analyze how different methods affect the collaboration between agents and measure differences in knowledge and reasoning tasks.Our results show that voting protocols improve performance by 13.2% in reasoning tasks and consensus protocols by 2.8% in knowledge tasks compared to other decision protocols.Increasing the number of agents improves performance, while more discussion rounds before voting reduce it.To improve decisionmaking by increasing answer diversity, we propose two new methods, All-Agents Drafting (AAD) and Collective Improvement (CI).Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI.This work demonstrates the importance of decisionmaking in multi-agent debates beyond scaling.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper systematically compares seven decision protocols in multi-agent debate while changing only the protocol, across knowledge and reasoning tasks. It finds voting protocols better for reasoning tasks and consensus protocols better for knowledge tasks, and proposes All-Agents Drafting (AAD) and Collective Improvement (CI) to improve answer diversity.
Key findings
- Voting protocols improve reasoning-task performance by 13.2%, while consensus protocols improve knowledge-task performance by 2.8%, compared with other decision protocols. Consensus beats the voting average by 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA; voting beats the consensus average by 13.1% on SQuAD 2.0, 0.2% on StrategyQA, and 26.4% on MuSR.
- Increasing the number of agents improves performance, while more discussion rounds before voting reduce it. Voting-based protocols require 3.38 rounds on average and consensus-based protocols 1.42 rounds; approval voting often fails to reach an agreed solution in 59% of cases.
- AAD improves task performance by up to 3.3% and CI by up to 7.4%. On StrategyQA, mean accuracy is 58.3% for baseline, 62.8% for AAD, 65.7% for CI, 59.4% for critical response generator, and 51.9% for reasoning response generator.
- On SQuAD 2.0, voting achieves higher overall F1 (56.7%) than consensus (43.6%), especially for answerable samples, but consensus is more effective for unanswerable samples, with stricter consensus methods producing higher scores.
Methods
- Multi-agent debates use three automatically generated expert personas, Llama 3 8B and 70B models, and a consistent setup varying only the decision protocol. Seven protocols are tested: four voting (simple, ranked, cumulative, approval) and three consensus (majority, supermajority, unanimity).
- Evaluation uses six datasets: knowledge tasks MMLU, MMLU-Pro, and GPQA; reasoning tasks SQuAD 2.0, StrategyQA, and MuSR. Because of computational constraints, the authors use sampled subsets with a 95% confidence level and 5% margin of error, run each condition three times, and report standard deviation.
- Analyses compare accuracy/F1 across protocols, rounds needed to decide, number of agents and discussion rounds, and answer diversity via cosine similarity between SBERT embeddings of agent answers.
Limitations
- The paper states that multi-agent debates are computationally expensive, requiring hundreds of forward passes per model, and that sampled subsets can introduce variance; they control for this with 95% confidence sampling and three independent runs.
- The paper states that agents often converged on similar responses despite efforts to improve answer diversity, and that prompt design and persona selection are relevant but a dedicated persona ablation was omitted due to computational cost and prior evidence.
- Evident from the text: the scaling and diversity experiments are mainly conducted on StrategyQA, the main protocol comparison uses Llama 3 8B with Llama 3 70B results in the appendix, and the use of dataset subsets and three runs limits broader generalization.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
27 of 39 cited sources checked · 6 supported · 5 flagged · accuracy not yet measured
- Study design and conductcitation
The study uses a consistent setup to quantify changes for which only the decision mechanism is responsible.
“In contrast to prior work, which varies the decision protocol concurrently to other parameters (Yang et al., 2024b;Yin et al., 2023), we use a consistent setup to quantify changes for which only the decision mechanism is responsible.”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Yang et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Yin et al., 2023 source
The citation attaches to the description of prior work, not to the citing authors’ claim about their own setup. In the paper’s ‘Termination Condition’ analysis (page unavailable), the authors compare two termination rules separately within each of four communication paradigms. That is contrary to the assertion that this work varies the decision protocol concurrently with other parameters. (LLM, conf=0.78)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
- Premisecitation
A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution.
“A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution (Zheng et al., 2023).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Zheng et al., 2023 source
Section 3.1 (p. 4) says comparing two answers can reveal subtle differences that single-answer grading may miss. That supports a narrower point about judging responses, but the paper does not establish that generating a more diverse set of answers makes it easier to select a correct solution than relying on one agent’s solution. (LLM, conf=0.78)
Experimental — accuracy not yet measured
- Premisecitation
Prior evidence suggests limited impact of persona ablation on task performance in similar setups.
“A dedicated persona ablation was omitted due to significant computational cost and prior evidence suggesting limited impact on task performance in similar setups (Zheng et al., 2024).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Zheng et al., 2024 source
- Study design and conductcitation
The sampling strategy ensures a 95% confidence level with a 5% margin of error.
“This follows approaches used by Yin et al. (2023); Chen et al. (2023); Becker (2024) and ensures a 95% confidence level with a 5% margin of error:”
Quote in the paperSubstantial claim3 of 3 sources checked- cites Yin et al. (2023) source
Section B, “Datasets and Evaluation Metrics” (page not provided), says the authors used all test samples in the main experiment and capped analysis runs at 1,000 samples for cost and rate-limit reasons. Section 4.4 discusses model-answer confidence, not statistical sampling confidence. Searches for a 95% confidence level, a 5% margin of error, and a sampling strategy found no passage addressing the claim. (LLM, conf=0.75)
Experimental — accuracy not yet measured - cites Chen et al. (2023) source
Appendix B.1 says the authors reported results using 100 test samples, but gives budget constraints and previous practice as the reasons. Appendix B.4 discusses confidence scores assigned by LLMs, not statistical confidence in a sample. Our copy does not say that a sampling strategy ensures a 95% confidence level or a 5% margin of error. (LLM, conf=0.75)
Experimental — accuracy not yet measured - cites Becker (2024) source
- Findingown data
CI improves task performance by up to 7.4%.
“Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI.”
- Also stated in: § ACCURACY Baseline All Agents Drafting · plain
“On average, CI performs 7.4% better than the baseline.”
- Also stated in: § 5 Conclusion · plain
“AAD improved performance by up to 3.3%, and CI by up to 7.4% over multi-agent debate baselines, and 6.1% and 10.2% over single model CoT baseline respectively.”
Quote in the paperSubstantial claim - Also stated in: § ACCURACY Baseline All Agents Drafting · plain
- Findingown data
On average, voting-based protocols require 3.38 rounds to reach a decision.
“On average, voting-based protocols require 3.38 rounds and consensus-based protocols 1.42 rounds to reach a decision, indicating that consensus is more suitable for applications that require quick decision-making as they use less test-time compute.”
Quote in the paperSubstantial claim - Findingown data
More discussion rounds before voting reduce performance.
“Increasing the number of agents improves performance, while more discussion rounds before voting reduce it.”
- Also stated in: § 4.2 Number of Agents and Discussion Rounds · hedged
“A linear regression, visualized by the dotted line, indicates that increasing discussion rounds slightly reduces accuracy.”
- Also stated in: § 5 Conclusion · plain
“Increasing the number of agents in the discussion improved task performance, while increasing the number of discussion rounds before voting decreased performance.”
Quote in the paperSubstantial claim - Also stated in: § 4.2 Number of Agents and Discussion Rounds · hedged
- Findingown data
Consensus protocols perform better on knowledge-based tasks with average improvements of 2.3% on MMLU over the voting average.
“Notably, consensus protocols perform better on knowledge-based tasks with average improvements of 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA over the voting average.”
Quote in the paperSubstantial claim - Findingown data
Increasing the number of agents improves performance.
“Increasing the number of agents improves performance, while more discussion rounds before voting reduce it.”
- Also stated in: § 4.2 Number of Agents and Discussion Rounds · hedged
“Here, a linear regression indicates that increasing the number of agents leads to a slight upward trend in accuracy, suggesting that larger groups improve task performance.”
- Also stated in: § 5 Conclusion · plain
“Increasing the number of agents in the discussion improved task performance, while increasing the number of discussion rounds before voting decreased performance.”
Quote in the paperSubstantial claim - Also stated in: § 4.2 Number of Agents and Discussion Rounds · hedged
- Findingown data
Consensus protocols improve performance by 2.8% in knowledge tasks compared to other decision protocols.
“Our results show that voting protocols improve performance by 13.2% in reasoning tasks and consensus protocols by 2.8% in knowledge tasks compared to other decision protocols.”
- Also stated in: § Agents Discuss · plain
“Our experiments show that consensus protocols perform better in knowledge tasks, improving performance by 2.8%, and voting protocols perform better in reasoning tasks, improving results by 13.2%.”
- Also stated in: § 5 Conclusion · plain
“In comparison, consensus outperforms voting in knowledge tasks by up to 2.8%, because it improves fact-checking by requiring at least the agreement of the majority of agents.”
Quote in the paperSubstantial claim - Also stated in: § Agents Discuss · plain
- Findingown data
AAD improves task performance by up to 3.3%.
“Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI.”
- Also stated in: § ACCURACY Baseline All Agents Drafting · plain
“AAD increases the performance on average by 3.3% over the baseline.”
- Also stated in: § 5 Conclusion · plain
“AAD improved performance by up to 3.3%, and CI by up to 7.4% over multi-agent debate baselines, and 6.1% and 10.2% over single model CoT baseline respectively.”
Quote in the paperSubstantial claim - Also stated in: § ACCURACY Baseline All Agents Drafting · plain
- Findingown data
Multi-agent debate with consensus and voting decision protocols outperforms the CoT baseline.
“Overall, multi-agent debate with consensus and voting decision protocols outperforms the CoT baseline.”
Quote in the paperSubstantial claim - Findingown data
Consensus protocols perform better on knowledge-based tasks with average improvements of 4.9% on MMLU-Pro over the voting average.
“Notably, consensus protocols perform better on knowledge-based tasks with average improvements of 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA over the voting average.”
Quote in the paperSubstantial claim - Findingown data
Voting protocols improve performance by 13.2% in reasoning tasks compared to other decision protocols.
“Our results show that voting protocols improve performance by 13.2% in reasoning tasks and consensus protocols by 2.8% in knowledge tasks compared to other decision protocols.”
- Also stated in: § Agents Discuss · plain
“Our experiments show that consensus protocols perform better in knowledge tasks, improving performance by 2.8%, and voting protocols perform better in reasoning tasks, improving results by 13.2%.”
- Also stated in: § 5 Conclusion · plain
“Our findings show that voting performs well on reasoning tasks, outperforming consensus by up to 13.2%, and outperforming a single CoT baseline by 10.4%.”
Quote in the paperSubstantial claim - Also stated in: § Agents Discuss · plain
- Findingown data
The strongest improvements from AAD are for simple and ranked voting.
“The strongest improvements are for simple and ranked voting.”
Quote in the paperNot a substantial claim - Findingreasoning
Voting outperforms a single CoT baseline by 10.4% on reasoning tasks.
“Our findings show that voting performs well on reasoning tasks, outperforming consensus by up to 13.2%, and outperforming a single CoT baseline by 10.4%.”
Quote in the paperSubstantial claim - Findingown data
The critical response generator does not provide a reliable improvement.
“The results show that the critical response generator does not provide a reliable improvement, while the reasoning response generator even leads to a decrease in task performance.”
Quote in the paperSubstantial claim - Findingown data
Stricter consensus methods (e.g., unanimity consensus) produce even higher scores on unanswerable samples.
“However, consensus is more effective for unanswerable samples, and stricter consensus methods (e.g., unanimity consensus) produce even higher scores.”
Quote in the paperSubstantial claim - Findingown data
The different challenge scenarios do not significantly impact the quality of the challenged answers.
“Overall, the different challenge scenarios (Only Solution, Discussion History, and Additional Information) do not significantly impact the quality of the challenged answers, as they all perform very similarly.”
Quote in the paperNot a substantial claim - Findingown data
If answer diversity increases, task performance increases too.
“We observe a general trend that if the answer diversity increases, task performance increases, too.”
Quote in the paperSubstantial claim - Findingown data
Agents can detect irrelevant solutions with a 99% challenge rate.
“Agents can detect irrelevant solutions (99% challenge rate) and incorrect solutions (75% challenge rate) with high accuracy.”
Quote in the paperNot a substantial claim - Findingown data
There are no major effects on knowledge tasks (MMLU and MMLU-Pro) from challenging answers.
“There are no major effects on knowledge tasks (MMLU and MMLU-Pro).”
Quote in the paperNot a substantial claim - Findingnone given
Most conversations reach a decision before they are terminated after five turns.
“Interestingly, most conversations reach a decision before they are terminated after five turns.”
Quote in the paperNot a substantial claim - Findingown data
On average, consensus-based protocols require 1.42 rounds to reach a decision.
“On average, voting-based protocols require 3.38 rounds and consensus-based protocols 1.42 rounds to reach a decision, indicating that consensus is more suitable for applications that require quick decision-making as they use less test-time compute.”
Quote in the paperSubstantial claim - Findingown data
The reasoning response generator leads to a decrease in task performance.
“The results show that the critical response generator does not provide a reliable improvement, while the reasoning response generator even leads to a decrease in task performance.”
Quote in the paperSubstantial claim - Findingown data
Voting achieves a higher overall F1-score (56.7%) than consensus (43.6%) on SQuAD 2.0.
“Voting achieves a higher overall F1-score (56.7%) than consensus (43.6%), especially for answerable samples.”
Quote in the paperSubstantial claim - Findingown data
CI improved performance by 10.2% over single model CoT baseline.
“AAD improved performance by up to 3.3%, and CI by up to 7.4% over multi-agent debate baselines, and 6.1% and 10.2% over single model CoT baseline respectively.”
Quote in the paperSubstantial claim - Findingreasoning
Approval voting often leads to no agreed solution (59% of cases).
“Approval voting is the only protocol that often leads to no agreed solution (59% of cases) due to the agreeableness of the agents.”
Quote in the paperSubstantial claim - Findingown data
Multi-agent approaches require approximately five times the compute resources for consensus protocols compared to the CoT baseline in our setup.
“While multi-agent approaches generally require more computation than a single agent (approximately five times the compute resources for consensus protocols and ten times for voting protocols compared to the CoT baseline in our setup), the choice between protocols offers distinct advantages beyond simply scaling resources.”
Quote in the paperNot a substantial claim - Findingown data
Consensus is more effective for unanswerable samples on SQuAD 2.0.
“However, consensus is more effective for unanswerable samples, and stricter consensus methods (e.g., unanimity consensus) produce even higher scores.”
Quote in the paperSubstantial claim - Findingown data
Agents can detect incorrect solutions with a 75% challenge rate.
“Agents can detect irrelevant solutions (99% challenge rate) and incorrect solutions (75% challenge rate) with high accuracy.”
Quote in the paperSubstantial claim - Findingown data
Providing additional information has no noticeable effect compared to just providing the solution.
“Providing additional information has no noticeable effect compared to just providing the solution.”
Quote in the paperSubstantial claim - Findingown data
AAD improved performance by 6.1% over single model CoT baseline.
“AAD improved performance by up to 3.3%, and CI by up to 7.4% over multi-agent debate baselines, and 6.1% and 10.2% over single model CoT baseline respectively.”
Quote in the paperSubstantial claim - Findingown data
Agents are less likely to challenge the final result when the discussion history is provided (10% decrease).
“We find that agents are less likely to challenge the final result when the discussion history is provided (10% decrease), as seen in Figure 6.”
Quote in the paperSubstantial claim - Findingown data
Voting can be easily tricked by generating answers that seem to be correct but have some incorrect statements.
“We found voting can be easily tricked by generating answers that seem to be correct but have some incorrect statements (see Section G.2).”
Quote in the paperSubstantial claim - Findingown data
Multi-agent approaches require approximately ten times the compute resources for voting protocols compared to the CoT baseline in our setup.
“While multi-agent approaches generally require more computation than a single agent (approximately five times the compute resources for consensus protocols and ten times for voting protocols compared to the CoT baseline in our setup), the choice between protocols offers distinct advantages beyond simply scaling resources.”
Quote in the paperNot a substantial claim - Findingown data
For the StrategyQA dataset, the decrease in challenge rate when discussion history is provided is 60%.
“For the StrategyQA dataset, this change is more prominent, with a 60% decrease in the challenge rate.”
Quote in the paperNot a substantial claim - Findingown data
The approval decision protocol only achieves first-round voting in about 27% of cases.
“In contrast, the approval decision protocol only achieves this in ∼ 27% of the cases.”
Quote in the paperSubstantial claim - Findingown data
Variations in agent response styles (critical, reasoning only) show decision protocol dependent effects.
“Furthermore, Section 4.3 analyzes variations in agent response styles (critical, reasoning only), showing decision protocol dependent effects.”
Quote in the paperNot a substantial claim - Findingnone given
Agents often converged on similar responses despite efforts to improve answer diversity.
“Despite efforts to improve answer diversity, agents often converged on similar responses, suggesting that more advanced techniques to encourage independent solutions are needed in the future.”
Quote in the paperNot a substantial claim - Findingreasoning
Many improvements by Llama 3 8B are quite small, except where Llama 3 70B also outperforms the CoT baseline.
“Many of the improvements by the Llama 3 8B model are quite small, except for the ones where the Llama 3 70B model also outperforms the CoT baseline.”
Quote in the paperSubstantial claim - Findingown data
Compared to Llama 3 8B, Llama 3 70B performs much better overall.
“Compared to the results of the Llama 3 8B model, the larger Llama 3 70B model performs much better overall, as seen in Figure 11.”
Quote in the paperNot a substantial claim - Findingown data
Most voting decision protocols are able to vote for a final answer already in the first round in which they are allowed to vote.
“Most of the voting decision protocols are able to vote for a final answer already in the first round in which they are allowed to vote.”
Quote in the paperNot a substantial claim - Findingown data
Solution counting generally underperforms compared to consensus- and voting-based decision protocols.
“While solution counting is a feasible method, it generally underperforms compared to consensus-and voting-based decision protocols.”
Quote in the paperSubstantial claim - Findingown data
In the study's multi-agent debate experiments, Approval Voting consistently fails to reach a voting decision.
“Approval Voting is left out as it consistently fails to reach a voting decision as described in Section 4.1.”
Quote in the paperSubstantial claim - Interpretationown data
Changes in decision protocols lead to markedly different results among tasks.
“We show that changes in decision protocols lead to markedly different results among tasks.”
- Also stated in: § 4.1 Performance of Decision Protocols · plain
“The effectiveness of decision protocols depends on the nature of the task.”
- Also stated in: § 4.1 Performance of Decision Protocols · plain
“The detailed analysis of SQuAD 2.0 further supports the finding that decision protocol effectiveness is task-dependent.”
Quote in the paperSubstantial claim - Also stated in: § 4.1 Performance of Decision Protocols · plain
- Interpretationown data
The decision protocol used can significantly impact the overall performance of multi-agent debates.
“Both experiments reveal that the decision protocol used can significantly impact the overall performance of multi-agent debates.”
Quote in the paperSubstantial claim - Interpretationown data
Consensus is more suitable for applications that require quick decision-making as they use less test-time compute.
“On average, voting-based protocols require 3.38 rounds and consensus-based protocols 1.42 rounds to reach a decision, indicating that consensus is more suitable for applications that require quick decision-making as they use less test-time compute.”
Quote in the paperSubstantial claim - Interpretationreasoning
Consensus's good performance in knowledge tasks is because it mitigates individual agent errors by requiring multiple agents to agree on the same statement before reaching a conclusion.
“One reason for the good performance of consensus in knowledge tasks is that it mitigates individual agent errors by requiring multiple agents to agree on the same statement before reaching a conclusion.”
Quote in the paperSubstantial claim - Interpretationreasoning
Consensus improves fact-checking by requiring at least the agreement of the majority of agents.
“In comparison, consensus outperforms voting in knowledge tasks by up to 2.8%, because it improves fact-checking by requiring at least the agreement of the majority of agents.”
Quote in the paperSubstantial claim - Interpretationreasoning
The consensus method has a distinct advantage by incorporating a safeguard of repeated checks across agents to find small errors.
“The consensus method has a distinct advantage by incorporating a safeguard of repeated checks across agents to find these small errors.”
Quote in the paperSubstantial claim - Interpretationnone given
Observed performance improvements may not always scale proportionally with the increased computational resources.
“Observed performance improvements, although positive, may not always scale proportionally with the increased computational resources.”
Quote in the paperNot a substantial claim - Interpretationown data
Approval voting's failure to reach a solution is due to the agreeableness of the agents.
“Approval voting is the only protocol that often leads to no agreed solution (59% of cases) due to the agreeableness of the agents.”
Quote in the paperSubstantial claim - Interpretationreasoning
The critical and restrictive prompting style degrades the quality of the discussion by forcing a specific answer style that is not always beneficial.
“The critical and restrictive prompting style degrades the quality of the discussion by forcing a specific answer style that is not always beneficial.”
Quote in the paperSubstantial claim - Interpretationreasoning
Consensus is less susceptible to manipulation or errors by any single agent.
“This approach is less susceptible to manipulation or errors by any single agent.”
Quote in the paperSubstantial claim - Interpretationreasoning
Voting protocols might outperform consensus in reasoning tasks because they allow the exploration of multiple reasoning paths.
“Voting protocols might outperform consensus in reasoning tasks because they allow the exploration of multiple reasoning paths.”
Quote in the paperSubstantial claim - Interpretationreasoning
More restrictive voting decision protocols can reach a decision more easily because a tie is less likely.
“Therefore, more restrictive voting decision protocols can reach a decision more easily, as a tie is less likely.”
Quote in the paperSubstantial claim - Interpretationreasoning
The big performance gain from the smaller model cannot be observed with Llama 3 70B.
“Therefore, the big performance gain from the smaller model cannot be observed here.”
Quote in the paperSubstantial claim - Study design and conductcitation
The small subset of each dataset still represents the dataset effectively.
“Because multi-agent discussions are expensive, we use a small subset of each dataset that still represents the dataset effectively.”
Quote in the paperSubstantial claim - Premisereasoning
Multi-agent debates are computationally expensive because they require a message from each agent in each round, quickly leading to hundreds of forward passes per model.
“Multi-agent debates are computationally expensive because they require a message from each agent in each round, quickly leading to hundreds of forward passes per model.”
Quote in the paperSubstantial claim - Interpretationreasoning
Using multiple agents can mitigate hallucination because it is less likely that two agents hallucinate the same things.
“This can be mitigated by using multiple agents because it is less likely that two agents hallucinate the same things.”
Quote in the paperSubstantial claim - Interpretationcitation
If discussions continue too long, they can drift away from the original task, which reduces task performance.
“If these discussions continue too long, they can drift away from the original task, which reduces task performance.”
Quote in the paperNot a substantial claim
- May combine two claims. 37
- Classified as probably not a substantial claim. 38
May combine two claims.
PremisecitationPrior work focuses on a single class of decision protocols without systematic comparison, partly because many parameters change between experiments.
“In summary, prior work focuses on a single class of decision protocols without systematic comparison, partly because many parameters change between experiments (Yin et al., 2023;Yang et al., 2024b;Chen et al., 2023).”
Quote in the paperSubstantial claim2 of 3 sources checked- cites Yin et al., 2023 source
The paper makes the contrary point about its own work: the Conclusion says it develops four communication paradigms, and the Experimental Setups compare their performance across tasks. Figure 6 also compares two termination rules. Our copy does not attribute a lack of comparison to changing parameters between experiments. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured - cites Yang et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Chen et al., 2023 source
The paper does not describe a lack of systematic comparison as a reason for its approach. Instead, its Experimental Setup describes matching the average number of LLM calls for a fair comparison, and Table 12 compares different voting strategies. The cited paper therefore provides a contrary example to the claim about decision protocols; it does not attribute the alleged lack of comparison to changing parameters between experiments. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationExchange-of-Thought (Yin et al., 2023) employs a consensus-based approach but does not compare consensus to other decision protocols.
“Exchangeof-Thought (Yin et al., 2023) employs a consensusbased approach, where agents iteratively refine answers through discussion in reasoning tasks, but they do not compare consensus to other decision protocols.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Yin et al., 2023 source
The paper does use majority-consensus termination, but it explicitly compares that approach with consistent-output termination. In the “Termination Condition” results (page unavailable), it reports higher performance for majority consensus across all four communication paradigms, contrary to the claim that it does not compare decision protocols. (LLM, conf=0.93)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationThe idea of independent answer generation has been previously explored in different multi-agent settings, but these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.
“While the idea of independent answer generation has been previously explored in different multi-agent settings (Du et al., 2023b;Liang et al., 2024), these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Du et al., 2023b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Liang et al., 2024 source
The Multi-Agent Debate Framework describes debaters speaking in a fixed order, with each response based on the previous debate history; the negative debater is instructed to disagree with the affirmative debater’s points. That is contrary to citing this paper as an example of independent answer generation. The paper analyzes debate features such as disagreement and adaptive stopping, but does not establish the claimed independent-generation step. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
PremisecitationBecker (2024) suggests that extended discussions may lead to decreased accuracy because of agents drifting away from the original task.
“Becker (2024) suggests that extended discussions may lead to decreased accuracy because of agents drifting away from the original task.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Becker, 2024 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationReConcile (Chen et al., 2023) integrates a hybrid voting and consensus approach by iteratively refining answers through confidence-weighting until consensus is reached.
“ReConcile (Chen et al., 2023) integrates a hybrid voting and consensus approach by iteratively refining answers through confidenceweighting until consensus is reached.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Chen et al., 2023 source
Section 4 describes multi-round discussion in which agents revise their answers and a confidence-weighted vote produces the team answer after each round. It says discussion stops when consensus is reached or after a maximum number of rounds, so consensus is not guaranteed; answer refinement also occurs through discussion, not confidence-weighting alone. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationResearch by Yin et al. (2023) suggests that increasing the number of rounds and agents participating in a multi-agent debate is beneficial because of test-time compute scaling.
“Research by Yin et al. (2023) suggests that increasing the number of rounds and agents participating in a multi-agent debate is beneficial because of testtime compute scaling.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Yin et al., 2023 source
The Round Analysis says EoT permits additional exchanges for difficult questions, with some SVAMP cases requiring more than five rounds. But it does not show that increasing rounds generally improves performance. The Limitations section discusses the possibility of involving more models, not a demonstrated benefit from adding agents; the paper also does not attribute such a benefit to test-time compute scaling. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPersonas can unlock more knowledge for the LLM on a specific topic.
“This can unlock more knowledge for the LLM on a specific topic (Kim et al., 2024).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kim et al., 2024 source
Section 2.1 says a domain-specific persona related to a question can improve an LLM’s performance. The paper does not establish that personas unlock more topic-specific knowledge; it measures reasoning performance, and Section 4.2 reports that personas sometimes worsen it. (LLM, conf=0.80)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationYin et al. (2023) found the number of personas (three) to be the most efficient.
“While the number of personas can vary, Yin et al. (2023) found this number to be the most efficient.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Yin et al., 2023 source
The D EoT Prompts section describes three personas—Kitty, Ben, and Peter—and §5.1 says the experiments used three models by default. The Cost Analysis compares EoT with other methods, but does not establish that three personas are the most efficient number. (LLM, conf=0.88)
Experimental — accuracy not yet measured
May combine two claims.
PremisecitationThis principle is also key to self-consistency approaches and explains why multiple-choice tests are often easier than open-ended ones.
“This principle is also key to self-consistency approaches (Wang et al., 2023) and explains why multiple-choice tests are often easier than openended ones.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Wang et al., 2023 source
The cited paper’s Experiments and Analysis sections discuss Solo Performance Prompting, but the passages found do not address self-consistency or compare multiple-choice with open-ended tests. Searches for both points found no supporting passage. The citing sentence also names Wang et al. (2023), whereas the supplied source is a 2024 paper. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationPresenting the task each round mitigates potential drift from the topic of the discussion.
“The prompt language is neutral, and the task is presented each round to mitigate the potential drift from the topic of the discussion (Becker, 2024).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Becker, 2024 source
The discussion prompt in Appendix G.2 includes the task instruction for each agent contribution, supporting the claim that the task is presented each round. Section 5.1.1 describes problem drift during discussions, but the paper does not say that repeating the task mitigates it or that this is why the prompt repeats it. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationEach dataset was tested three times to obtain a standard deviation, ensuring reliable and robust performance estimates.
“Each dataset was tested three times to obtain a standard deviation of the results (Reimers and Gurevych, 2017;Chen et al., 2023;Becker, 2024), ensuring reliable and robust performance estimates across multiple evaluations.”
Quote in the paperNot a substantial claim3 of 3 sources checked- cites Reimers and Gurevych, 2017 source
Section 5, “Robust Model Evaluation,” says that rerunning nondeterministic approaches and comparing score distributions is essential for sound conclusions; the conclusion also discusses studying the standard deviation of scores. The paper does not recommend or establish three tests per dataset as sufficient for reliable estimates. Its reported rerun counts include 86 and 41 for two systems (Section 3), so the specific three-test procedure is not supported by this citation. (LLM, conf=0.87)
Experimental — accuracy not yet measured - cites Chen et al., 2023 source
The Implementation Details section says the authors reported accuracy and standard deviation and conducted at least three runs for each experiment on the same test samples and prompts. It does not say every dataset was tested exactly three times or establish that these runs ensured reliable and robust estimates. (LLM, conf=0.83)
Experimental — accuracy not yet measured - cites Becker, 2024 source
The experimental-sampling passage (section “373(×5)”) says the author reported standard deviations across repeated dataset experiments, supporting that part of the claim. It specifies five runs, not three; the Limitations section (6.3) likewise describes five runs on randomized subsets. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThis resembles the effect seen in self-consistency, where increasing the number of sampled answers improves task performance.
“This resembles the effect seen in self-consistency (Wang et al., 2023), where increasing the number of sampled answers improves task performance.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Wang et al., 2023 source
The Experiments section and Table 5 discuss Solo Performance Prompting and compare task scores, but do not address self-consistency or whether sampling more answers improves performance. Searches for self-consistency and multiple sampled answers found no passage in our copy that speaks to the claimed effect; the table’s N values refer to numbers of trivia questions, not sampled answers. Page numbers were not available in the indexed passages. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
PremisecitationUsing personas is important as it generates agents with expertise in different domains, incorporating diverse viewpoints for the final decision.
“Using personas is important as it generates agents with expertise in different domains, incorporating diverse viewpoints for the final decision.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Kim et al., 2024 source
Sections 3.1 and 4.8 describe generating a question-appropriate persona, with different personas appearing for different questions. Sections 3.2 and 3.3 describe a final decision informed by two perspectives, but these are one persona-based solver and one neutral solver—not multiple agents with expertise in different domains contributing to the same decision. No page numbers are available in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationHuang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.
“This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
Quote in the paperNot a substantial claim2 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCurrent research shows that discussion protocols have little impact on downstream task performance.
“Current research by Yin et al. (2023) and Becker (2024) shows that discussion protocols have little impact on downstream task performance.”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites Yin et al. (2023) source
Table 1 (section labelled COMMUNICATION PROMPTS; page unavailable) shows similar average scores for the four communication paradigms, from 87.20 to 87.50. But the paper also says different paradigms perform best on different tasks, so the broad claim that discussion protocols have little impact on downstream performance is not fully supported. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites Becker (2024) source
The Results (§5.1.1; page unavailable) show that some discussion paradigms perform similarly, but also say the report paradigm significantly improves performance on ethical questions. The claim that protocols have little impact on downstream performance is therefore too broad: Becker finds that their effect depends on the task. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThis drift has also been observed by Becker (2024).
“This has also been observed by Becker (2024) and an example can be seen in Section G.3.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Becker (2024) source
May combine two claims.
Findingown dataConsensus protocols perform better on knowledge-based tasks with average improvements of 1.3% on GPQA over the voting average.
“Notably, consensus protocols perform better on knowledge-based tasks with average improvements of 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA over the voting average.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataAn additional experiment comparing solution counting to prompted decision protocols showed similar trends and had little impact on main findings.
“An additional experiment comparing solution counting (selecting the answer that is given the most) to our prompted decision protocols showed similar trends and had little impact on our main findings (see Section D).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataCumulative voting, supermajority, and unanimity consensus perform a bit worse with AAD.
“Cumulative voting, supermajority, and unanimity consensus perform a bit worse.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataConsensus protocols often reached a decision faster than voting protocols, particularly on knowledge tasks that also achieved superior performance.
“Notably, consensus protocols often reached a decision faster than voting protocols, particularly on knowledge tasks that also achieved superior performance (Figure 2).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataChallenging answers with a changed final score has no positive effect, especially on reasoning tasks, where it leads to worse solutions in 3% of cases.
“It can be seen that this has no positive effect, especially on the reasoning tasks (StrategyQA and SQuAD 2.0), where it leads to worse solutions in 3% of cases.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAgents often simply vote for all answers as they do not have to decide, leading to many ties.
“The agents often simply vote for all answers as they do not have to decide, leading to many ties (see Section A.2 for details).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAgents are agreeable and often only improve the answer from the first agent without proposing an idea based on their own expertise.
“Throughout our experiments, we observe that agents are agreeable and often only improve the answer from the first agent without proposing an idea based on their own expertise.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataApproval voting and majority have no real improvement with AAD but are within the standard deviation of the baseline.
“Approval voting and majority have no real improvement but are within the standard deviation of the baseline.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataVoting excelled on reasoning tasks despite potentially needing more rounds and compute.
“Conversely, voting excelled on reasoning tasks despite potentially needing more rounds and compute.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenLlama 3 70B does not fail to follow the prompt for the MuSR baseline and consensus-based decision protocols.
“This model does not fail to follow the prompt for the MuSR baseline and consensus-based decision protocols.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenDecision protocols tend to behave similarly in terms of rounds needed to create a final answer, independent of task and model.
“The decision protocols tend to behave similarly in terms of rounds needed to create a final answer, independent of task and model.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataConsensus decision protocols require only one to two rounds to reach consensus and still achieve higher task performance on the MMLU dataset.
“The consensus decision protocols require only one to two rounds to reach consensus and still achieve a higher task performance because these results are based on the MMLU dataset.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataSimple voting has the highest agreement rate, but cumulative and ranked voting only need another round in a few cases.
“Simple voting has the highest agreement rate, but also cumulative and ranked voting only need in a few cases another round.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenInitial explorations with more diverse prompt structures did not yield consistent improvements over the simpler, reproducible design presented.
“Initial explorations with more diverse prompt structures did not yield consistent improvements over the simpler, reproducible design presented.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataMost results are a bit better than the baseline, but multi-agent discussions only outperform the CoT baseline in a few cases.
“Most of the results are a bit better than the baseline, but the multi-agent discussions are only in a few cases able to outperform the CoT baseline.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenLlama 3 70B has a much higher baseline for task performance, making it more difficult to improve baseline results.
“In general, Llama 3 70B has a much higher baseline for task performance, making it more difficult to improve baseline results.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone givenSQuAD 2.0 and StrategyQA had the largest performance gains, even outperforming the CoT baseline, similar to the smaller model.
“SQuAD 2.0 and StrategyQA had the largest performance gains, even outperforming the CoT baseline, similar to the results from the smaller model.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenOverall, the results were markedly higher than what could be explained by the standard deviation.
“Overall, the results were markedly higher than what could be explained by the standard deviation.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAbout 14% need another round and the rest is canceled after the fifth round.
“About 14% need another round and the rest is canceled after the fifth round.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenIn the G.3 Bad Voting Discussion example, the agents were tricked by one agent who proposed a solution not provided in the context.
“In this discussion, the agents were tricked by one agent who came up with a solution not provided in the context.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataConsensus methods outperform solution counting on knowledge-based tasks (MMLU-Pro, GPQA), and voting notably improves performance on the reasoning task (StrategyQA).
“In particular, consensus methods outperform solution counting on knowledge-based tasks (MMLU-Pro, GPQA), and voting notably improves performance on the reasoning task (Strat-egyQA).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningDecision-making is central to multi-agent processes.
“Thus, we argue that decisionmaking is central to multi-agent processes.”
- Also stated in: § Abstract · boosted
“This work demonstrates the importance of decisionmaking in multi-agent debates beyond scaling.”
Quote in the paperNot a substantial claim- Also stated in: § Abstract · boosted
May combine two claims.
InterpretationreasoningWith the correct choice, multi-agent debates can be made more error-resistant by using consensus decision protocols and more explorative by using voting decision protocols.
“This demonstrates that with the correct choice, multi-agent debates can be made more error-resistant by using consensus decision protocols and more explorative by using voting decision protocols.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningAAD and CI achieve higher answer diversity and therefore higher task performance.
“AAD and CI achieve higher answer diversity and, therefore, higher task performance, but the two response generators struggle to provide reliable results and may even reduce task performance.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductown dataThe decision protocol is the only variable changed in the systematic comparison.
“We change only one variable at a time -the decision protocol -to analyze how different methods affect the collaboration between agents and measure differences in knowledge and reasoning tasks.”
- Also stated in: § 4.1 Performance of Decision Protocols · plain
“This experiment systematically compares a set of seven decision protocols, changing only the decision protocol used and nothing else.”
Quote in the paperSubstantial claim- Also stated in: § 4.1 Performance of Decision Protocols · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationFinding collective solutions markedly impacts human decision-making.
“Finding collective solutions markedly impacts human decision-making (Jones, 1994).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Jones, 1994 source
May combine two claims.
Interpretationown dataMore debate rounds harm task performance, suggesting the expected benefits of self-refinement may be unreliable and differ from task to task.
“These findings further support that more debate rounds harm task performance, suggesting the expected benefits of self-refinement may be unreliable and also differ from task to task.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe reasoning response generator decreases answer diversity, which results in a drop in task performance.
“The reasoning response generator decreases answer diversity, which results in a drop in task performance.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningTask performance is calculated on a subset of samples with three runs and standard deviation is reported.
“Because of computational constraints, we calculate the task performance on a subset of samples with three runs and report their standard deviation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationParticipants can vote for a less preferred but more viable alternative to block an undesired outcome, leading to outcomes that do not reflect the collective will of the group.
“For example, participants can vote for a less preferred but more viable alternative to block an undesired outcome, leading to outcomes that do not reflect the collective will of the group (List, 2022).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites List, 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
PremisecitationYang et al. (2024b,a) introduce multiple voting protocols but do not compare the performance of these voting decision protocols across different tasks.
“Yang et al. (2024b,a) introduce multiple voting protocols (e.g., approval voting) but do not compare the performance of these voting decision protocols across different tasks, focussing more on a comparison of how humans vote compared to LLMs.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Yang et al., 2024b,a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenEach condition in the scaling experiment is evaluated across three independent runs.
“Both experiments use the simple voting decision protocol, and each condition is evaluated across three independent runs.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataOur methods enhance answer diversity and reveal a connection between answer diversity and task performance.
“Our methods enhance answer diversity and reveal a connection between answer diversity and task performance.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataConsensus excels in knowledge tasks, and voting works better for reasoning tasks.
“Our results suggest that consensus excels in knowledge tasks, and voting works better for reasoning tasks.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationThree rounds of discussion allow for reasonably strong improvements considering computing resources.
“In the default setting, they have to discuss for three rounds, as current research such as Du et al. (2023b) shows that this allows for reasonable strong improvements considering computing resources.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Du et al. (2023b)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
PremisecitationLarger models tend to hallucinate less, reducing the hallucination effect for Llama 3 70B.
“Larger models tend to hallucinate less, reducing this effect for the Llama 3 70B model (Li et al., 2024).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Li et al. (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenBy changing the way an agent is prompted, a lot of performance can be gained or lost.
“By changing the way an agent is prompted, a lot of performance can be gained or lost.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenReasoning-based datasets (StrategyQA, MuSR, SQuAD 2.0) emphasize multistep reasoning and textual comprehension.
“StrategyQA, MuSR, and SQuAD 2.0.These emphasize multistep reasoning and textual comprehension.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenPrompt design and persona selection are relevant parameters for multi-agent debate.
“While this study focuses on decision protocols, we acknowledge that prompt design and persona selection are also relevant parameters.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationSmaller models are more likely to hallucinate, which reduces task performance.
“As Li et al. (2024) showed, smaller models are more likely to hallucinate, which reduces task performance.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Li et al. (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningMulti-agent discussions require specific problem structures, or else agents just talk about the same results for multiple rounds and agree with each other.
“This can be taken as evidence that these multi-agent discussions require specific problem structures, or else the agents are just talking about the same results for multiple rounds and agreeing with each other.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationcitationEncouraging agents to be more critical may reduce sycophancy.
“Encouraging them to be more critical may reduce this.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenKnowledge-based datasets (MMLU, MMLU-Pro, GPQA) still require some reasoning and domain knowledge.
“MMLU, MMLU-Pro, and GPQA.These still require some reasoning and domain knowledge.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningUsing sampled subsets of the datasets can lead to some variance.
“Because of the high computational cost and the range of decision protocols and tasks in our work, we used sampled subsets of the datasets, which can lead to some variance.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningModels like to agree with each other, so they tend to vote for many answers, often leading to a tie.
“This happens because these models like to agree with each other, and therefore they tend to vote for many of the answers, which often leads to a tie.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationLLMs can show some form of sycophancy, which is not helpful for a constructive discussion.
“Some studies have shown that LLMs can show some form of sycophancy, which is not helpful for a constructive discussion (Sharma et al., 2023).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Sharma et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationWang et al. (2024a) shows increasing the number of agents may have positive effects.
“Wang et al. (2024a) shows increasing the number of agents may have positive effects.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wang et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe Critical Response Generator encourages agents to critically assess prior answers and propose new solutions, countering sycophantic tendencies.
“The Critical Response Generator encourages agents to critically assess prior answers and propose new solutions, countering sycophantic tendencies (Sharma et al., 2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Sharma et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoningThe study controlled for variance by sampling with a 95% confidence level and calculating the standard deviation of three independent runs.
“To control for that variance, we sampled with a 95% confidence level and calculated the standard deviation of three independent runs.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisereasoningMulti-agent discussions produce multiple results for the same problem because each agent has its own reasoning and ideas.
“Multiagent discussions produce multiple results for the same problem because each agent has its own reasoning and ideas on how to solve the problem.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationown dataThe solution counting ablation reinforces the main findings and illustrates the added value of decision protocols.
“Thus, this ablation reinforces our main findings and illustrates the added value of decision protocols.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Prompt design has a marked impact on multi-agent debate.
“Prompt design has a marked impact on multi-agent debate.”
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Not confirmedz = 1.96reported p = .5 · recomputed p = .050
“Z = 1.96, p = .5”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
AAD improvement does not match the table's accuracies
MajorTable 1 reports baseline accuracy of 58.3% and All draft accuracy of 62.8%, a gain of 4.5 percentage points, not the reported 3.3 points.
Reported 3.3 · recomputed 4.5
“| Mean Accuracy | 58.3% | 62.8% | 65.7% | 59.4% | 51.9% | | Delta Baseline | 0.0% | 3.3% | 7.4% | 1.1% | -6.4% |”
Cumulative voting has conflicting point limits
MinorThe methodology permits each agent to allocate up to 25 points, whereas the supplied cumulative-voting prompt instructs agents to distribute 10 points.
“cumulative voting -each agent is allowed to divide up to 25 points between all solutions.”
“You are tasked with distributing 10 points among the provided solutions based on the given task.”
Consensus sometimes takes more than two rounds
MinorThe text says consensus protocols require only one to two rounds, but Table 2 records nonzero shares of consensus decisions in later rounds, including 12.67%, 3.50%, and 2.67% for unanimity.
“The consensus decision protocols require only one to two rounds to reach consensus”
“| Unanimity 59.50% 21.67% 12.67% 3.50% | 2.67% | 54.2 ± 1.0 | | |”
Cites
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1× · see citing works
“To quantify if these methods increase answer diversity, we calculate the cosine similarity between the SBERT embeddings (Reimers and Gurevych, 2019) of the agent’s answers and correlate it with changes in task performance.”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 1× · see citing works
“…of decision-making protocols within multi-agent debates on knowledge tasks (i.e., MMLU (Hendrycks et al., 2021), MLLU-Pro (Wang et al., 2024b), GPQA (Rein et al., 2023)) and reasoning tasks (i.e., SQuAD 2.0 (Rajpurkar et al., 2018), StrategyQA (Geva et al., 2021), MuSR (Sprague et al., 2023)).”
InfluentialJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1× · see citing works
“A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent’s solution (Zheng et al., 2023).”
George, M. (1998) · Nursing Standard · doi:10.7748/ns.12.26.24.s36 · Cited by 1,615
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2019 · Cited by 755 · see citing works
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1× · see citing works
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
Reimers, N., & Gurevych, I. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1035 · Cited by 463 · cited 1× · see citing works
“Each dataset was tested three times to obtain a standard deviation of the results (Reimers and Gurevych, 2017; Chen et al., 2023; Becker, 2024), ensuring reliable and robust performance estimates across multiple evaluations.”
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.19118 · Cited by 320 · cited 1× · see citing works
“While the idea of independent answer generation has been previously explored in different multi-agent settings (Du et al., 2023b; Liang et al., 2024), these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.”
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 13× · see citing works
“The data in Table 2 shows the number of turns that are needed for each decision protocol to reach a final decision for the MMLU dataset.”
InfluentialEncouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 274 · see citing works
Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?Preprint
Wang, Q., Wang, Z., Su, Y., Tong, H.-H., & Song, Y.-Q. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.18272 · Cited by 243 · cited 1× · see citing works
“Wang et al. (2024a) shows increasing the number of agents may have positive effects.”
<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · cited 11× · see citing works
“SQuAD 2.0 and StrategyQA had the largest performance gains, even outper-forming the CoT baseline, similar to the results from the smaller model.”
InfluentialMuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Sprague, Z., Ye, X., Bostrom, K., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.16049 · Cited by 241 · cited 6×
“This model does not fail to follow the prompt for the MuSR baseline and consensus-based decision protocols.”
InfluentialSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · cited 1× · see citing works
“This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
Jiang, H., Zhang, X., Cao, X., Breazeal, C., Roy, D. K., & Kabbara, J. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.229 · Cited by 201 · cited 1× · see citing works
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
ExpeL: LLM Agents Are Experiential Learners
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2023) · arXiv · doi:10.48550/arxiv.2308.10144 · Cited by 179 · cited 1× · see citing works
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
Meta-Prompting: Enhancing Language Models with Task-Agnostic ScaffoldingPreprint
Suzgun, M., & Kalai, A. (2024) · arXiv.org · doi:10.48550/arxiv.2401.12954 · Cited by 159 · cited 1× · see citing works
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 132 · see citing works
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.01574 · Cited by 131 · cited 2× · see citing works
“This study systematically quantifies the effectiveness of decision-making protocols within multi-agent debates on knowledge tasks (i.e., MMLU (Hendrycks et al., 2021), MLLU-Pro (Wang et al., 2024b), GPQA (Rein et al., 2023)) and reasoning tasks (i.e., SQuAD 2.0 (Rajpurkar et al., 2018), StrategyQA (Geva et al., 2021),…”
InfluentialMMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3018 · Cited by 131 · see citing works
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 96 · cited 1× · see citing works
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 86 · see citing works
Pushing up to a point: Assertiveness and effectiveness in leadership and interpersonal dynamics
Ames, D. R. (2009) · Research in Organizational Behavior · doi:10.1016/j.riob.2009.06.010 · Cited by 82 · cited 1× · see citing works
“This could also involve examining personas within this hierarchical structure to investigate whether dominant or affectionate leaders are more effective in leading discussions (Ames, 2009).”
Zheng, M., Pei, J., Logeswaran, L., Lee, M., & Jurgens, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.888 · Cited by 82 · see citing works
Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?
Wang, Q., Wang, Z., Su, Y., Tong, H., & Song, Y. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.331 · Cited by 80 · see citing works
Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., & Ji, H. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.15 · Cited by 74 · cited 1× · see citing works
“The default setting for the framework is to prompt a LLM and ask for a persona relevant to the given task (Wang et al., 2024c).”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Chen, J. C.-Y., Saha, S., & Bansal, M. (2023) · arXiv · doi:10.48550/arxiv.2309.13007 · Cited by 64 · cited 5× · see citing works
“Multi-agent debates for task solving have shown promise in recent research, but they rely on a varying choice of decision protocols, hindering systematic comparison (Yin et al., 2023; Chen et al., 2023).”
InfluentialReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Chen, J., Saha, S., & Bansal, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.381 · Cited by 64 · see citing works
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10671 · Cited by 63 · see citing works
ExpertPrompting: Instructing Large Language Models to be Distinguished ExpertsPreprint
Xu, B., An, Y., Lin, J., Wang, Q., Zhou, C., Zhang, Y., & Mao, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14688 · Cited by 58 · cited 1× · see citing works
“In the following, we explain the multi-agent environment, decision protocols, response generators, and datasets used in our experiments.”
LLM Voting: Human Choices and AI Collective Decision-Making
Yang, J. C., Dailisan, D. N., Korecki, M., Hausladen, C. I., & Helbing, D. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31758 · Cited by 43 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · cited 1× · see citing works
“This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.12022 · Cited by 30 · cited 8× · see citing works
“3% on GPQA over the voting average.”
InfluentialAre Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection
Wahle, J. P., Ruas, T., Meuschke, N., & Gipp, B. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00065 · Cited by 29 · cited 1× · see citing works
“Solutions are generated with a different model than Llama 3, to remove any bias that might occur if the same model used to detect the wrong answer also generated it (Wahle et al., 2021).”
Self-citationExchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26 · see citing works
MALLM: Multi-Agent Large Language Models FrameworkPreprint
Becker, J., Kaesberg, L. B., Bauer, N., Wahle, J. P., Ruas, T., & Gipp, B. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2509.11656 · Cited by 25 · cited 1× · see citing works
“To better understand the different modules, we take a closer look at each component and what role it plays in creating multi-agent discussions.”
Self-citationYang, J. C., Hausladen, C. I., Peters, D., Pournaras, E., Fricker, R. H., & Helbing, D. (2024) · Digital Government Research and Practice · doi:10.1145/3665332 · Cited by 24 · see citing works
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2312.01823 · Cited by 21 · cited 10×
“We extend the majority consensus of Yin et al. (2023) by higher agreement levels, i.e., supermajority (66%), unanimity (100%).”
InfluentialCiteAssist: A System for Automated Preprint Citation and BibTeX Generation
Kaesberg, L. B., Ruas, T., Wahle, J. P., et al. (2024) · SDP · doi:10.48550/arxiv.2407.03192 · Cited by 14
Self-citationWiley Series in Probability and Statistics
Thompson, J. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118109656.scard · Cited by 12
A comparison of consensus and voting in public decision making
Jones, B. (1994) · Negotiation Journal · doi:10.1007/bf02184175 · Cited by 11 · cited 1× · see citing works
“History, sociology, and psychology have long demonstrated how different decision-making processes influence collective outcomes (Jones, 1994; List, 2022).”
Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection
Wahle, J. P., Ruas, T., Meuschke, N., & Gipp, B. (2021) · Figshare · doi:10.5281/zenodo.4621402 · Cited by 7 · see citing works
Self-citationMulti-Agent Large Language Models for Conversational Task-Solving
Becker, J. (2024) · arXiv · doi:10.48550/arxiv.2410.22932 · Cited by 3 · cited 7×
“Becker (2024) suggests that extended discussions may lead to decreased accuracy because of agents drifting away from the original task.”
InfluentialSelf-citationKim, J., Yang, N., & Jung, K. (2024) · arXiv · doi:10.48550/arxiv.2408.08631 · Cited by 2 · cited 2×
“Each discussion consists of three automatically generated expert personas for the task following Kim et al. (2024).”
AI Usage Cards: Responsibly Reporting AI-generated ContentPreprint
Wahle, J. P., Ruas, T., Mohammad, S. M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.03886 · Cited by 1 · see citing works
Self-citationCiteAssist: A System for Automated Preprint Citation and BibTeX Generation
Kaesberg, L. B., Ruas, T., Wahle, J. P., et al. (2024) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2024.sdp-1.10
Self-citationCochran, W. G. (1953) · doi:10.2307/3706682
Flagged first, then by citations
Research works printed but not linked: 1
- Christian List. 2022. Social Choice Theory. In The Stanford Encyclopedia of Philosophy, winter 2022 edition.
Cited by
Sorted by citations
AI Agents in Clinical Medicine: A Systematic ReviewPreprint
Gorenshtein, A., Omar, M., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025) · medRxiv · doi:10.1101/2025.08.22.25334232 · Cited by 59 · see citing works
Large language models in 6G from standard to on-device networks
Zou, H., Zhao, Q., Lasaulce, S., Zhang, C., Tian, Y., Bariah, L., Bader, F., & Debbah, M. A. (2026) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-025-00239-6 · Cited by 4 · see citing works
“They typically require in the order of 2–4 rounds, meaning approximately 4–12 message exchanges per task, with gains saturating beyond about 4 rounds 89 .”
Auditing medical multi-agent AI reveals risks of false consensus
Zhu, Y., Gu, L., Wang, Z., Sang, H., Sui, D., Tang, W., Mi, L., Wang, Y., Gao, J., Yao, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.10185 · Cited by 2 · see citing works
A Role-Aware Multi-Agent Framework for Financial Education QA
Zhu, A., & Du, Y. (2025) · International Conference on AI in Finance · doi:10.1145/3768292.3770345 · Cited by 2 · see citing works
“Multi-agent systems in question answering were proposed to combine the roles of several specialized agents to vote and reach a better answer [12].”
The Fragility of Moral Judgment in Large Language Models
van Nuenen, T., & Sachdeva, P. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806447 · Cited by 1 · see citing works
“This corroborates prior evidence that protocol choices are consequential [23, 38] and challenges the assumption that benchmarks measure stable dispositions.”
The Physics of Getting Elected: Financial Dynamics in Local Council Elections in BrazilPreprint
Marques, A. S., Santos, A., da Rosa, R. C., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5382183 · Cited by 1 · see citing works
Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
Wunderlich, F. V., Kaesberg, L. B., Wahle, J. P., Ruas, T., & Gipp, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) · doi:10.18653/v1/2026.acl-srw.1
“Our result indicates that scaling rounds can negatively affect performance, aligning with prior work and resembling limitations of other sequential scaling approaches, such as self-refinement (Kaesberg et al., 2025a; Becker et 2025b).”
Self-citationWojarnik, G. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.505
Model-Specific Echo Chamber Formation Patterns in Repeated LLM Agent Networks
Jo, S., Kim, J., Lee, S., & Seok, H. (2026) · doi:10.1145/3816713.3816738
“More recent work further suggests that the decision-making method itself in multi-agent debate can influence the results, and that sycophancy during the interaction process may lower the reliability and efficiency of the debate [5, 7, 8].”
A Systematic Evaluation of Decision Mechanisms in Multi-agent LLMs for Misinformation Detection
Lee, Y., Huang, Y., & Chu, T.-H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30044-7_6
Consensus-Gated Execution: A Multi-Agent LLM Architecture for Autonomous Cryptocurrency Trading
Bajgora, A., Kulakov, A., & Merovcı, F. (2026) · Electronics · doi:10.3390/electronics15153453
Vatsal, S., Dubey, H., & Bajaj, A. (2026) · IEEE Access · doi:10.1109/access.2026.3720396
“Studies that isolate decision protocols illustrate how rule choices shape collaboration outcomes [91].”
Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted On
Yao, Y., Yao, Y., Fan, X., Gao, J., Wang, J., Zhang, M., Ravi, S., & Joe‐Wong, C. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818655
“…LLM-as-a-judge monitoring [17, 73], structured role-play [54], deep-research style iterative decomposition [67], explicit human-agent-computer interaction loops [36], training-free or post-hoc error correction loops [74], and ensemble-style decision aggregation such as voting [30] and debate [19].”
ReGAP: Evidence-Aligned Conversational Question Generation via Reasoning-Guided Action Planning
Wang, W., He, L., Zhou, P., & Zheng, W. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3838193
“It can be obtained through expert annotation or approximated via LLM-based multi-agent debate [35].”
TRACE-QA: Task-routed constraint elimination for auditable multi-agent question answering
Lu, J., Chen, H., Zhu, Y., Xu, J., & Xu, A. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01141-3 · Cited by 1 · see citing works
“Recent multi-agent frameworks further emphasize diversity and protocol design, showing that both inter-agent interaction structure and final aggregation rule matter substantially (Liang et al. 2023; Chen et al. 2024; Kaesberg et al. 2025).”
Belz, J. H., Kim, K., Rukzio, E., et al. (2026) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3828158.3838235
Gladkikh, K. I., Шабалин, А. М., & Zakharov, A. A. (2026) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon69876.2026.11707165
“…ensemble parameter: majority voting, unanimity consensus, supermajority consensus, and other schemes each perform best depending on the task [11] This matters for binary vulnerability classification in particular, since the choice of aggregation strategy directly sets the trade-off between…”
Multi-agent discussion gains less when dissent is withheld
Mansuri, C. S., Wang, X., Li, M., Acton, B., Eckardt, R., Patel, D., & Kojaku, S. (2026) · arXiv · doi:10.48550/arxiv.2609.38324
“Some studies show that discussion, often run as multi-agent debate, outperforms standalone reasoning or voting (Du et al., 2024; Liang et al., 2024; Chen et al., 2024), whereas others report no effect (Smit et al., 2024; Wang et al., 2024b; Kaesberg et al., 2025).”
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
Emergence of Biased Consensus in Multi-Agent LLM Debates
Okawa, M. (2026) · arXiv · doi:10.48550/arxiv.2608.02827
“…strategies (e.g., dynamic agent recruitment in AgentVerse (Chen et al., 2023) and debate-mode switching in ChatEval (Chan et al., 2023)), and decision-making protocols such as voting, consensus (Kaesberg et al., 2025; Choi et al., 2025a), and confidence-weighted aggregation (Chen et al., 2024).”
- Authors
- Kaesberg, L. B., Becker, J., Wahle, J. P., Ruas, T., & Gipp, B.
- Affiliations
- University of Göttingen (DE)
- Landeskriminalamt Nordrhein-Westfalen (DE)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2025
- Published
- 2025
- Pages
- 11640-11671
- OpenAlex
- W4412888276
- Cited by
- 14
- References
- 47
- Citations by year
- 2026: 11 · 2025: 4
- FWCI
- 15.37
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- GRO (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- multi-agent debate, decision procedures, majority voting, consensus algorithms, agent cooperation, number of agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review