Cited by 20Multi-Agent Systems and NegotiationTopic ModelingLogic, Reasoning, and Knowledge

Voting or Consensus? Decision-Making in Multi-Agent Debate

Open access
Kaesberg, Lars Benedikt, Becker, Jonas, Wahle, Jan Philip, Ruas, Terry, Gipp, Bela·2025·Findings of the Association for Computational Linguistics: ACL 2025

Published version of a preprint

Abstract

Much of the success of multi-agent debates depends on carefully choosing the right parameters.The decision-making protocol stands out as it can highly impact final model answers, depending on how decisions are reached.Systematic comparison of decision protocols is difficult because many studies alter multiple discussion parameters beyond the protocol.So far, it has been largely unknown how decisionmaking influences different tasks.This work systematically evaluates the impact of seven decision protocols (e.g., majority voting, unanimity consensus).We change only one variable at a time -the decision protocol -to analyze how different methods affect the collaboration between agents and measure differences in knowledge and reasoning tasks.Our results show that voting protocols improve performance by 13.2% in reasoning tasks and consensus protocols by 2.8% in knowledge tasks compared to other decision protocols.Increasing the number of agents improves performance, while more discussion rounds before voting reduce it.To improve decisionmaking by increasing answer diversity, we propose two new methods, All-Agents Drafting (AAD) and Collective Improvement (CI).Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI.This work demonstrates the importance of decisionmaking in multi-agent debates beyond scaling.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper systematically compares seven decision protocols in multi-agent debate while changing only the protocol, across knowledge and reasoning tasks. It finds voting protocols better for reasoning tasks and consensus protocols better for knowledge tasks, and proposes All-Agents Drafting (AAD) and Collective Improvement (CI) to improve answer diversity.

Key findings

  • Voting protocols improve reasoning-task performance by 13.2%, while consensus protocols improve knowledge-task performance by 2.8%, compared with other decision protocols. Consensus beats the voting average by 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA; voting beats the consensus average by 13.1% on SQuAD 2.0, 0.2% on StrategyQA, and 26.4% on MuSR.
  • Increasing the number of agents improves performance, while more discussion rounds before voting reduce it. Voting-based protocols require 3.38 rounds on average and consensus-based protocols 1.42 rounds; approval voting often fails to reach an agreed solution in 59% of cases.
  • AAD improves task performance by up to 3.3% and CI by up to 7.4%. On StrategyQA, mean accuracy is 58.3% for baseline, 62.8% for AAD, 65.7% for CI, 59.4% for critical response generator, and 51.9% for reasoning response generator.
  • On SQuAD 2.0, voting achieves higher overall F1 (56.7%) than consensus (43.6%), especially for answerable samples, but consensus is more effective for unanswerable samples, with stricter consensus methods producing higher scores.

Methods

  • Multi-agent debates use three automatically generated expert personas, Llama 3 8B and 70B models, and a consistent setup varying only the decision protocol. Seven protocols are tested: four voting (simple, ranked, cumulative, approval) and three consensus (majority, supermajority, unanimity).
  • Evaluation uses six datasets: knowledge tasks MMLU, MMLU-Pro, and GPQA; reasoning tasks SQuAD 2.0, StrategyQA, and MuSR. Because of computational constraints, the authors use sampled subsets with a 95% confidence level and 5% margin of error, run each condition three times, and report standard deviation.
  • Analyses compare accuracy/F1 across protocols, rounds needed to decide, number of agents and discussion rounds, and answer diversity via cosine similarity between SBERT embeddings of agent answers.

Limitations

  • The paper states that multi-agent debates are computationally expensive, requiring hundreds of forward passes per model, and that sampled subsets can introduce variance; they control for this with 95% confidence sampling and three independent runs.
  • The paper states that agents often converged on similar responses despite efforts to improve answer diversity, and that prompt design and persona selection are relevant but a dedicated persona ablation was omitted due to computational cost and prior evidence.
  • Evident from the text: the scaling and diversity experiments are mainly conducted on StrategyQA, the main protocol comparison uses Llama 3 8B with Llama 3 70B results in the appendix, and the use of dataset subsets and three runs limits broader generalization.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2025.findings-acl.606

Cites

44 of 49 references verified0 citation errors · 5 not checkable

Statistics

Claims the paper makes

27 of 39 cited sources checked · 6 supported · 5 flagged · accuracy not yet measured

  • Study design and conductcitation

    The study uses a consistent setup to quantify changes for which only the decision mechanism is responsible.

    “In contrast to prior work, which varies the decision protocol concurrently to other parameters (Yang et al., 2024b;Yin et al., 2023), we use a consistent setup to quantify changes for which only the decision mechanism is responsible.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Yang et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yin et al., 2023 source

      The citation attaches to the description of prior work, not to the citing authors’ claim about their own setup. In the paper’s ‘Termination Condition’ analysis (page unavailable), the authors compare two termination rules separately within each of four communication paradigms. That is contrary to the assertion that this work varies the decision protocol concurrently with other parameters. (LLM, conf=0.78)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution.

    “A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution (Zheng et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al., 2023 source

      Section 3.1 (p. 4) says comparing two answers can reveal subtle differences that single-answer grading may miss. That supports a narrower point about judging responses, but the paper does not establish that generating a more diverse set of answers makes it easier to select a correct solution than relying on one agent’s solution. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    Prior evidence suggests limited impact of persona ablation on task performance in similar setups.

    “A dedicated persona ablation was omitted due to significant computational cost and prior evidence suggesting limited impact on task performance in similar setups (Zheng et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al., 2024 source
  • Study design and conductcitation

    The sampling strategy ensures a 95% confidence level with a 5% margin of error.

    “This follows approaches used by Yin et al. (2023); Chen et al. (2023); Becker (2024) and ensures a 95% confidence level with a 5% margin of error:”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Yin et al. (2023) source

      Section B, “Datasets and Evaluation Metrics” (page not provided), says the authors used all test samples in the main experiment and capped analysis runs at 1,000 samples for cost and rate-limit reasons. Section 4.4 discusses model-answer confidence, not statistical sampling confidence. Searches for a 95% confidence level, a 5% margin of error, and a sampling strategy found no passage addressing the claim. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites Chen et al. (2023) source

      Appendix B.1 says the authors reported results using 100 test samples, but gives budget constraints and previous practice as the reasons. Appendix B.4 discusses confidence scores assigned by LLMs, not statistical confidence in a sample. Our copy does not say that a sampling strategy ensures a 95% confidence level or a 5% margin of error. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites Becker (2024) source
  • Findingown data

    CI improves task performance by up to 7.4%.

    “Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI.”
    • Also stated in: § ACCURACY Baseline All Agents Drafting · plain
      “On average, CI performs 7.4% better than the baseline.”
    • Also stated in: § 5 Conclusion · plain
      “AAD improved performance by up to 3.3%, and CI by up to 7.4% over multi-agent debate baselines, and 6.1% and 10.2% over single model CoT baseline respectively.”
    Quote in the paperSubstantial claim
  • May combine two claims. 37
  • Classified as probably not a substantial claim. 38
  • May combine two claims.

    Premisecitation

    Prior work focuses on a single class of decision protocols without systematic comparison, partly because many parameters change between experiments.

    “In summary, prior work focuses on a single class of decision protocols without systematic comparison, partly because many parameters change between experiments (Yin et al., 2023;Yang et al., 2024b;Chen et al., 2023).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Yin et al., 2023 source

      The paper makes the contrary point about its own work: the Conclusion says it develops four communication paradigms, and the Experimental Setups compare their performance across tasks. Figure 6 also compares two termination rules. Our copy does not attribute a lack of comparison to changing parameters between experiments. (LLM, conf=0.83)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites Yang et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chen et al., 2023 source

      The paper does not describe a lack of systematic comparison as a reason for its approach. Instead, its Experimental Setup describes matching the average number of LLM calls for a fair comparison, and Table 12 compares different voting strategies. The cited paper therefore provides a contrary example to the claim about decision protocols; it does not attribute the alleged lack of comparison to changing parameters between experiments. (LLM, conf=0.83)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • May combine two claims.

    Premisecitation

    Exchange-of-Thought (Yin et al., 2023) employs a consensus-based approach but does not compare consensus to other decision protocols.

    “Exchangeof-Thought (Yin et al., 2023) employs a consensusbased approach, where agents iteratively refine answers through discussion in reasoning tasks, but they do not compare consensus to other decision protocols.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Yin et al., 2023 source

      The paper does use majority-consensus termination, but it explicitly compares that approach with consistent-output termination. In the “Termination Condition” results (page unavailable), it reports higher performance for majority consensus across all four communication paradigms, contrary to the claim that it does not compare decision protocols. (LLM, conf=0.93)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • May combine two claims.

    Premisecitation

    The idea of independent answer generation has been previously explored in different multi-agent settings, but these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.

    “While the idea of independent answer generation has been previously explored in different multi-agent settings (Du et al., 2023b;Liang et al., 2024), these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Du et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liang et al., 2024 source

      The Multi-Agent Debate Framework describes debaters speaking in a fixed order, with each response based on the previous debate history; the negative debater is instructed to disagree with the affirmative debater’s points. That is contrary to citing this paper as an example of independent answer generation. The paper analyzes debate features such as disagreement and adaptive stopping, but does not establish the claimed independent-generation step. (LLM, conf=0.83)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim.

    Premisecitation

    Becker (2024) suggests that extended discussions may lead to decreased accuracy because of agents drifting away from the original task.

    “Becker (2024) suggests that extended discussions may lead to decreased accuracy because of agents drifting away from the original task.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Becker, 2024 source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ReConcile (Chen et al., 2023) integrates a hybrid voting and consensus approach by iteratively refining answers through confidence-weighting until consensus is reached.

    “ReConcile (Chen et al., 2023) integrates a hybrid voting and consensus approach by iteratively refining answers through confidenceweighting until consensus is reached.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chen et al., 2023 source

      Section 4 describes multi-round discussion in which agents revise their answers and a confidence-weighted vote produces the team answer after each round. It says discussion stops when consensus is reached or after a maximum number of rounds, so consensus is not guaranteed; answer refinement also occurs through discussion, not confidence-weighting alone. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Research by Yin et al. (2023) suggests that increasing the number of rounds and agents participating in a multi-agent debate is beneficial because of test-time compute scaling.

    “Research by Yin et al. (2023) suggests that increasing the number of rounds and agents participating in a multi-agent debate is beneficial because of testtime compute scaling.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Yin et al., 2023 source

      The Round Analysis says EoT permits additional exchanges for difficult questions, with some SVAMP cases requiring more than five rounds. But it does not show that increasing rounds generally improves performance. The Limitations section discusses the possibility of involving more models, not a demonstrated benefit from adding agents; the paper also does not attribute such a benefit to test-time compute scaling. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Personas can unlock more knowledge for the LLM on a specific topic.

    “This can unlock more knowledge for the LLM on a specific topic (Kim et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

      Section 2.1 says a domain-specific persona related to a question can improve an LLM’s performance. The paper does not establish that personas unlock more topic-specific knowledge; it measures reasoning performance, and Section 4.2 reports that personas sometimes worsen it. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Yin et al. (2023) found the number of personas (three) to be the most efficient.

    “While the number of personas can vary, Yin et al. (2023) found this number to be the most efficient.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Yin et al., 2023 source

      The D EoT Prompts section describes three personas—Kitty, Ben, and Peter—and §5.1 says the experiments used three models by default. The Cost Analysis compares EoT with other methods, but does not establish that three personas are the most efficient number. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • May combine two claims.

    Premisecitation

    This principle is also key to self-consistency approaches and explains why multiple-choice tests are often easier than open-ended ones.

    “This principle is also key to self-consistency approaches (Wang et al., 2023) and explains why multiple-choice tests are often easier than openended ones.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source

      The cited paper’s Experiments and Analysis sections discuss Solo Performance Prompting, but the passages found do not address self-consistency or compare multiple-choice with open-ended tests. Searches for both points found no supporting passage. The citing sentence also names Wang et al. (2023), whereas the supplied source is a 2024 paper. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Presenting the task each round mitigates potential drift from the topic of the discussion.

    “The prompt language is neutral, and the task is presented each round to mitigate the potential drift from the topic of the discussion (Becker, 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Becker, 2024 source

      The discussion prompt in Appendix G.2 includes the task instruction for each agent contribution, supporting the claim that the task is presented each round. Section 5.1.1 describes problem drift during discussions, but the paper does not say that repeating the task mitigates it or that this is why the prompt repeats it. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Each dataset was tested three times to obtain a standard deviation, ensuring reliable and robust performance estimates.

    “Each dataset was tested three times to obtain a standard deviation of the results (Reimers and Gurevych, 2017;Chen et al., 2023;Becker, 2024), ensuring reliable and robust performance estimates across multiple evaluations.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Reimers and Gurevych, 2017 source

      Section 5, “Robust Model Evaluation,” says that rerunning nondeterministic approaches and comparing score distributions is essential for sound conclusions; the conclusion also discusses studying the standard deviation of scores. The paper does not recommend or establish three tests per dataset as sufficient for reliable estimates. Its reported rerun counts include 86 and 41 for two systems (Section 3), so the specific three-test procedure is not supported by this citation. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Chen et al., 2023 source

      The Implementation Details section says the authors reported accuracy and standard deviation and conducted at least three runs for each experiment on the same test samples and prompts. It does not say every dataset was tested exactly three times or establish that these runs ensured reliable and robust estimates. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Becker, 2024 source

      The experimental-sampling passage (section “373(×5)”) says the author reported standard deviations across repeated dataset experiments, supporting that part of the claim. It specifies five runs, not three; the Limitations section (6.3) likewise describes five runs on randomized subsets. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This resembles the effect seen in self-consistency, where increasing the number of sampled answers improves task performance.

    “This resembles the effect seen in self-consistency (Wang et al., 2023), where increasing the number of sampled answers improves task performance.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source

      The Experiments section and Table 5 discuss Solo Performance Prompting and compare task scores, but do not address self-consistency or whether sampling more answers improves performance. Searches for self-consistency and multiple sampled answers found no passage in our copy that speaks to the claimed effect; the table’s N values refer to numbers of trivia questions, not sampled answers. Page numbers were not available in the indexed passages. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim.

    Premisecitation

    Using personas is important as it generates agents with expertise in different domains, incorporating diverse viewpoints for the final decision.

    “Using personas is important as it generates agents with expertise in different domains, incorporating diverse viewpoints for the final decision.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

      Sections 3.1 and 4.8 describe generating a question-appropriate persona, with different personas appearing for different questions. Sections 3.2 and 3.3 describe a final decision informed by two perspectives, but these are one persona-based solver and one neutral solver—not multiple agents with expertise in different domains contributing to the same decision. No page numbers are available in our copy. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.

    “This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Huang et al., 2023 source
    • cites Madaan et al., 2023 source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Current research shows that discussion protocols have little impact on downstream task performance.

    “Current research by Yin et al. (2023) and Becker (2024) shows that discussion protocols have little impact on downstream task performance.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Yin et al. (2023) source

      Table 1 (section labelled COMMUNICATION PROMPTS; page unavailable) shows similar average scores for the four communication paradigms, from 87.20 to 87.50. But the paper also says different paradigms perform best on different tasks, so the broad claim that discussion protocols have little impact on downstream performance is not fully supported. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Becker (2024) source

      The Results (§5.1.1; page unavailable) show that some discussion paradigms perform similarly, but also say the report paradigm significantly improves performance on ethical questions. The claim that protocols have little impact on downstream performance is therefore too broad: Becker finds that their effect depends on the task. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This drift has also been observed by Becker (2024).

    “This has also been observed by Becker (2024) and an example can be seen in Section G.3.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Becker (2024) source
  • May combine two claims.

    Findingown data

    Consensus protocols perform better on knowledge-based tasks with average improvements of 1.3% on GPQA over the voting average.

    “Notably, consensus protocols perform better on knowledge-based tasks with average improvements of 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA over the voting average.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    An additional experiment comparing solution counting to prompted decision protocols showed similar trends and had little impact on main findings.

    “An additional experiment comparing solution counting (selecting the answer that is given the most) to our prompted decision protocols showed similar trends and had little impact on our main findings (see Section D).”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Cumulative voting, supermajority, and unanimity consensus perform a bit worse with AAD.

    “Cumulative voting, supermajority, and unanimity consensus perform a bit worse.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Consensus protocols often reached a decision faster than voting protocols, particularly on knowledge tasks that also achieved superior performance.

    “Notably, consensus protocols often reached a decision faster than voting protocols, particularly on knowledge tasks that also achieved superior performance (Figure 2).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Challenging answers with a changed final score has no positive effect, especially on reasoning tasks, where it leads to worse solutions in 3% of cases.

    “It can be seen that this has no positive effect, especially on the reasoning tasks (StrategyQA and SQuAD 2.0), where it leads to worse solutions in 3% of cases.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Agents often simply vote for all answers as they do not have to decide, leading to many ties.

    “The agents often simply vote for all answers as they do not have to decide, leading to many ties (see Section A.2 for details).”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Agents are agreeable and often only improve the answer from the first agent without proposing an idea based on their own expertise.

    “Throughout our experiments, we observe that agents are agreeable and often only improve the answer from the first agent without proposing an idea based on their own expertise.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Approval voting and majority have no real improvement with AAD but are within the standard deviation of the baseline.

    “Approval voting and majority have no real improvement but are within the standard deviation of the baseline.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Voting excelled on reasoning tasks despite potentially needing more rounds and compute.

    “Conversely, voting excelled on reasoning tasks despite potentially needing more rounds and compute.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Llama 3 70B does not fail to follow the prompt for the MuSR baseline and consensus-based decision protocols.

    “This model does not fail to follow the prompt for the MuSR baseline and consensus-based decision protocols.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Decision protocols tend to behave similarly in terms of rounds needed to create a final answer, independent of task and model.

    “The decision protocols tend to behave similarly in terms of rounds needed to create a final answer, independent of task and model.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Consensus decision protocols require only one to two rounds to reach consensus and still achieve higher task performance on the MMLU dataset.

    “The consensus decision protocols require only one to two rounds to reach consensus and still achieve a higher task performance because these results are based on the MMLU dataset.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Simple voting has the highest agreement rate, but cumulative and ranked voting only need another round in a few cases.

    “Simple voting has the highest agreement rate, but also cumulative and ranked voting only need in a few cases another round.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Initial explorations with more diverse prompt structures did not yield consistent improvements over the simpler, reproducible design presented.

    “Initial explorations with more diverse prompt structures did not yield consistent improvements over the simpler, reproducible design presented.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Most results are a bit better than the baseline, but multi-agent discussions only outperform the CoT baseline in a few cases.

    “Most of the results are a bit better than the baseline, but the multi-agent discussions are only in a few cases able to outperform the CoT baseline.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingnone given

    Llama 3 70B has a much higher baseline for task performance, making it more difficult to improve baseline results.

    “In general, Llama 3 70B has a much higher baseline for task performance, making it more difficult to improve baseline results.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingnone given

    SQuAD 2.0 and StrategyQA had the largest performance gains, even outperforming the CoT baseline, similar to the smaller model.

    “SQuAD 2.0 and StrategyQA had the largest performance gains, even outperforming the CoT baseline, similar to the results from the smaller model.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Overall, the results were markedly higher than what could be explained by the standard deviation.

    “Overall, the results were markedly higher than what could be explained by the standard deviation.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    About 14% need another round and the rest is canceled after the fifth round.

    “About 14% need another round and the rest is canceled after the fifth round.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In the G.3 Bad Voting Discussion example, the agents were tricked by one agent who proposed a solution not provided in the context.

    “In this discussion, the agents were tricked by one agent who came up with a solution not provided in the context.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Consensus methods outperform solution counting on knowledge-based tasks (MMLU-Pro, GPQA), and voting notably improves performance on the reasoning task (StrategyQA).

    “In particular, consensus methods outperform solution counting on knowledge-based tasks (MMLU-Pro, GPQA), and voting notably improves performance on the reasoning task (Strat-egyQA).”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Decision-making is central to multi-agent processes.

    “Thus, we argue that decisionmaking is central to multi-agent processes.”
    • Also stated in: § Abstract · boosted
      “This work demonstrates the importance of decisionmaking in multi-agent debates beyond scaling.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    With the correct choice, multi-agent debates can be made more error-resistant by using consensus decision protocols and more explorative by using voting decision protocols.

    “This demonstrates that with the correct choice, multi-agent debates can be made more error-resistant by using consensus decision protocols and more explorative by using voting decision protocols.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    AAD and CI achieve higher answer diversity and therefore higher task performance.

    “AAD and CI achieve higher answer diversity and, therefore, higher task performance, but the two response generators struggle to provide reliable results and may even reduce task performance.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Study design and conductown data

    The decision protocol is the only variable changed in the systematic comparison.

    “We change only one variable at a time -the decision protocol -to analyze how different methods affect the collaboration between agents and measure differences in knowledge and reasoning tasks.”
    • Also stated in: § 4.1 Performance of Decision Protocols · plain
      “This experiment systematically compares a set of seven decision protocols, changing only the decision protocol used and nothing else.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Finding collective solutions markedly impacts human decision-making.

    “Finding collective solutions markedly impacts human decision-making (Jones, 1994).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jones, 1994 source
  • May combine two claims.

    Interpretationown data

    More debate rounds harm task performance, suggesting the expected benefits of self-refinement may be unreliable and differ from task to task.

    “These findings further support that more debate rounds harm task performance, suggesting the expected benefits of self-refinement may be unreliable and also differ from task to task.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    The reasoning response generator decreases answer diversity, which results in a drop in task performance.

    “The reasoning response generator decreases answer diversity, which results in a drop in task performance.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Task performance is calculated on a subset of samples with three runs and standard deviation is reported.

    “Because of computational constraints, we calculate the task performance on a subset of samples with three runs and report their standard deviation.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Participants can vote for a less preferred but more viable alternative to block an undesired outcome, leading to outcomes that do not reflect the collective will of the group.

    “For example, participants can vote for a less preferred but more viable alternative to block an undesired outcome, leading to outcomes that do not reflect the collective will of the group (List, 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites List, 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Yang et al. (2024b,a) introduce multiple voting protocols but do not compare the performance of these voting decision protocols across different tasks.

    “Yang et al. (2024b,a) introduce multiple voting protocols (e.g., approval voting) but do not compare the performance of these voting decision protocols across different tasks, focussing more on a comparison of how humans vote compared to LLMs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Yang et al., 2024b,a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each condition in the scaling experiment is evaluated across three independent runs.

    “Both experiments use the simple voting decision protocol, and each condition is evaluated across three independent runs.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationown data

    Our methods enhance answer diversity and reveal a connection between answer diversity and task performance.

    “Our methods enhance answer diversity and reveal a connection between answer diversity and task performance.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    Consensus excels in knowledge tasks, and voting works better for reasoning tasks.

    “Our results suggest that consensus excels in knowledge tasks, and voting works better for reasoning tasks.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    Three rounds of discussion allow for reasonably strong improvements considering computing resources.

    “In the default setting, they have to discuss for three rounds, as current research such as Du et al. (2023b) shows that this allows for reasonable strong improvements considering computing resources.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Du et al. (2023b)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Larger models tend to hallucinate less, reducing the hallucination effect for Llama 3 70B.

    “Larger models tend to hallucinate less, reducing this effect for the Llama 3 70B model (Li et al., 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Li et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    By changing the way an agent is prompted, a lot of performance can be gained or lost.

    “By changing the way an agent is prompted, a lot of performance can be gained or lost.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Reasoning-based datasets (StrategyQA, MuSR, SQuAD 2.0) emphasize multistep reasoning and textual comprehension.

    “StrategyQA, MuSR, and SQuAD 2.0.These emphasize multistep reasoning and textual comprehension.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Prompt design and persona selection are relevant parameters for multi-agent debate.

    “While this study focuses on decision protocols, we acknowledge that prompt design and persona selection are also relevant parameters.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    Smaller models are more likely to hallucinate, which reduces task performance.

    “As Li et al. (2024) showed, smaller models are more likely to hallucinate, which reduces task performance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Li et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Multi-agent discussions require specific problem structures, or else agents just talk about the same results for multiple rounds and agree with each other.

    “This can be taken as evidence that these multi-agent discussions require specific problem structures, or else the agents are just talking about the same results for multiple rounds and agreeing with each other.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Interpretationcitation

    Encouraging agents to be more critical may reduce sycophancy.

    “Encouraging them to be more critical may reduce this.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Knowledge-based datasets (MMLU, MMLU-Pro, GPQA) still require some reasoning and domain knowledge.

    “MMLU, MMLU-Pro, and GPQA.These still require some reasoning and domain knowledge.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Using sampled subsets of the datasets can lead to some variance.

    “Because of the high computational cost and the range of decision protocols and tasks in our work, we used sampled subsets of the datasets, which can lead to some variance.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Models like to agree with each other, so they tend to vote for many answers, often leading to a tie.

    “This happens because these models like to agree with each other, and therefore they tend to vote for many of the answers, which often leads to a tie.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    LLMs can show some form of sycophancy, which is not helpful for a constructive discussion.

    “Some studies have shown that LLMs can show some form of sycophancy, which is not helpful for a constructive discussion (Sharma et al., 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sharma et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Wang et al. (2024a) shows increasing the number of agents may have positive effects.

    “Wang et al. (2024a) shows increasing the number of agents may have positive effects.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Critical Response Generator encourages agents to critically assess prior answers and propose new solutions, countering sycophantic tendencies.

    “The Critical Response Generator encourages agents to critically assess prior answers and propose new solutions, countering sycophantic tendencies (Sharma et al., 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Sharma et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The study controlled for variance by sampling with a 95% confidence level and calculating the standard deviation of three independent runs.

    “To control for that variance, we sampled with a 95% confidence level and calculated the standard deviation of three independent runs.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Premisereasoning

    Multi-agent discussions produce multiple results for the same problem because each agent has its own reasoning and ideas.

    “Multiagent discussions produce multiple results for the same problem because each agent has its own reasoning and ideas on how to solve the problem.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The solution counting ablation reinforces the main findings and illustrates the added value of decision protocols.

    “Thus, this ablation reinforces our main findings and illustrates the added value of decision protocols.”
    Quote in the paperNot a substantial claim
67 of 67
  • The paper relies on this without support.

    Prompt design has a marked impact on multi-agent debate.

    “Prompt design has a marked impact on multi-agent debate.”
1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Not confirmedz = 1.96reported p = .5 · recomputed p = .050

    “Z = 1.96, p = .5”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • AAD improvement does not match the table's accuracies

    Major

    Table 1 reports baseline accuracy of 58.3% and All draft accuracy of 62.8%, a gain of 4.5 percentage points, not the reported 3.3 points.

    Reported 3.3 · recomputed 4.5

  • Cumulative voting has conflicting point limits

    Minor

    The methodology permits each agent to allocate up to 25 points, whereas the supplied cumulative-voting prompt instructs agents to distribute 10 points.

  • Consensus sometimes takes more than two rounds

    Minor

    The text says consensus protocols require only one to two rounds, but Table 2 records nonzero shares of consensus decisions in later rounds, including 12.67%, 3.50%, and 2.67% for unanimity.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Christian List. 2022. Social Choice Theory. In The Stanford Encyclopedia of Philosophy, winter 2022 edition.

Cited by

Sorted by citations

Authors
Kaesberg, L. B., Becker, J., Wahle, J. P., Ruas, T., & Gipp, B.
Affiliations
  • University of Göttingen (DE)
  • Landeskriminalamt Nordrhein-Westfalen (DE)
Venue
Findings of the Association for Computational Linguistics: ACL 2025
Published
2025
Pages
11640-11671
OpenAlex
W4412888276
Cited by
14
References
47
Citations by year
2026: 11 · 2025: 4
FWCI
15.37
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • arXiv (Cornell University) (submittedVersion)
  • GRO (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
multi-agent debate, decision procedures, majority voting, consensus algorithms, agent cooperation, number of agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review