CoChat
Sign Up
PreprintCited by 14Topic ModelingNatural Language Processing TechniquesMulti-Agent Systems and Negotiation

Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs

Open access
Smit, Andries, Duckworth, Paul, Grinsztajn, Nathan, Barrett, Thomas D., Pretorius, Arnu·2023·arXiv
DOI 10.48550/arxiv.2311.17371arXiv 2311.17371
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in large language models (LLMs) underscore their potential for responding to inquiries in various domains. However, ensuring that generative agents provide accurate and reliable answers remains an ongoing challenge. In this context, multi-agent debate (MAD) has emerged as a promising strategy for enhancing the truthfulness of LLMs. We benchmark a range of debating and prompting strategies to explore the trade-offs between cost, time, and accuracy. Importantly, we find that multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies, such as self-consistency and ensembling using multiple reasoning paths. However, when performing hyperparameter tuning, several MAD systems, such as Multi-Persona, perform better. This suggests that MAD protocols might not be inherently worse than other approaches, but that they are more sensitive to different hyperparameter settings and difficult to optimize. We build on these results to offer insights into improving debating strategies, such as adjusting agent agreement levels, which can significantly enhance performance and even surpass all other non-debate protocols we evaluated. We provide an open-source repository to the community with several state-of-the-art protocols together with evaluation scripts to benchmark across popular research datasets.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper benchmarks multi-agent debate (MAD) and other prompting strategies for multiple-choice QA, finding that current MAD systems do not reliably outperform non-debate methods such as Medprompt and self-consistency. However, MAD performance is highly sensitive to hyperparameters, and a new prompt-based “agreement intensity” modulation can substantially improve some debate systems.

Key findings

  • In best-configuration comparisons, Medprompt performed best overall on MedQA with 0.65 accuracy, while the best debating strategy, Society of Mind, reached 0.64; Multi-Persona was 0.58.
  • MAD protocols did not consistently outperform non-debate protocols on reasoning datasets; no protocol dominated on all datasets, and Medprompt seemed to perform best overall.
  • Hyperparameter tuning mattered: in medical datasets, every evaluated system outperformed single-agent methods except Multi-Persona; in non-medical datasets, single-agent methods surpassed all evaluated systems except Multi-Persona and Self-Consistency.
  • Modulating agent agreement improved Multi-Persona by ≈15% and Society of Mind by ≈5% on the USMLE subset, while ChatEval was hardly affected; applying 90% agreement intensity to Multi-Persona on full MedQA produced a new state-of-the-art result for GPT-3.

Methods

  • The authors benchmarked prompting strategies including Society of Mind, Multi-Persona, ChatEval, Self-Consistency, Ensemble Refinement, Medprompt, and single-agent baselines using GPT-3.5-turbo as the base agent.
  • Evaluation used seven datasets: MedQA (1,273 questions), PubMedQA (500), MMLU clinical topics (123), CosmosQA (500 subsampled), CIAR (50), GPQA (448), and Chess Short (1,000 games per subtask).
  • Analyses compared accuracy against cost, time, tokens, and API calls; used K-fold-style hyperparameter selection across medical and non-medical dataset groups; and tested agreement-intensity prompts plus GPT-4 and Mixtral 8x7B on MedQA.

Limitations

  • The paper states that API calls expose experiments to variable inference times and unforeseen model updates, and that large-scale API benchmarking incurs substantial financial and time costs, limiting experiment scale and creating barriers to replication and extension.
  • The paper notes that main experiments used GPT-3.5 for cost efficiency; GPT-4 tests were limited to a smaller subset, and hyperparameter transfer did not extend well to Mixtral 8x7B.
  • Evident from the text: several datasets were subsampled or restricted (e.g., CosmosQA 500 questions, Chess Short setting), and Medprompt was only partially implemented because its kNN component would require a training/test split.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.17371

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 6 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    One major reason why debate strategies can be effective is the ability of LLMs to adapt to additional information given in-context.

    “One major reason why debate strategies can be an effective tool is the ability of LLMs to adapt to additional information given in-context (Zhang et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zhang et al., 2023 source

      The Introduction and §3 (Preliminary) describe LLMs learning from examples supplied in a prompt, supporting the in-context adaptation part of the claim. The paper does not address debate strategies or establish that this ability makes them effective. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Study design and conductcitation

    API calls to a publicly available LLM expose the study to variable inference time calls and unforeseen model updates.

    “We utilize API calls to a publicly available LLM (Brown et al., 2020) which, whilst sufficient in the context of our investigation, exposes us to variable inference time calls and unforeseen model updates.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Brown et al., 2020 source
  • Findingown data

    Multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies such as self-consistency and ensembling using multiple reasoning paths.

    “Importantly, we find that multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies, such as self-consistency and ensembling using multiple reasoning paths.”
    Quote in the paperSubstantial claim
  • Findingnone given

    The single agent's performance, along with self-consistency, can be manipulated via prompts to achieve a high variance.

    “Somewhat concerning for the domain of medical Q&A, is that the single agent's performance, along with self-consistency, can be manipulated via prompts to achieve a high variance.”
    Quote in the paperNot a substantial claim
  • Findingown data

    When performing hyperparameter tuning, several MAD systems, such as Multi-Persona, perform better.

    “However, when performing hyperparameter tuning, several MAD systems, such as Multi-Persona, perform better.”
    Quote in the paperSubstantial claim
  • May combine two claims. 19
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 8

  • May combine two claims. The quote only partly states this claim.

    Findingcitation

    Medprompt, which does not use debate, performs best overall on MedQA and has lower costs associated with it.

    “However, the recently introduced Medprompt strategy (Nori et al., 2023), which does not use debate, performs the best overall.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Nori et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The step-by-step explanations were generated by a panel of qualified clinicians who identified the best examples and crafted the few-shot prompts as part of the Med-PaLM project.

    “The step-by-step explanations were generated by a panel of qualified clinicians who identified the best examples and crafted the few-shot prompts as part of the Med-PaLM project.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Singhal et al. (2023) source

      Section 3.3 (page not available) says the authors crafted chain-of-thought prompts with step-by-step explanations for Med-PaLM 2. It does not say a panel of qualified clinicians generated the explanations or selected the best examples; it says the few-shot prompts were reused from prior work, rather than crafted for this project. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The few-shot Q&A examples with explanations are provided for each medical dataset in Singhal et al. (2023).

    “The few-shot Q&A examples with explanations are provided for each medical dataset inSinghal et al. (2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Singhal et al. (2023) source

      The Additional information section (page unavailable) says Tables A.10–A.13 provide chain-of-thought prompts, and §3.3 explains that these add step-by-step explanations to few-shot examples. But the PubMedQA prompt in Table A.14 has no chain-of-thought explanations, while Table A.17 gives long-form datasets prompt templates rather than explained few-shot Q&A examples. The claim’s 'for each medical dataset' is therefore too broad. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Adjusting agent agreement levels can significantly enhance performance and even surpass all other non-debate protocols evaluated.

    “We build on these results to offer insights into improving debating strategies, such as adjusting agent agreement levels, which can significantly enhance performance and even surpass all other non-debate protocols we evaluated.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    A novel debate prompting strategy can modulate the level of agreement between agents during a debate and improve upon the state-of-the-art for Q&A for a given model class.

    “Concretely, we provide a novel debate prompting strategy able to modulate the level of agreement between agents during a debate and improve upon the state-of-the-art for Q&A for a given model class.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The highest performing debating strategy from those introduced in Section 2 is SoM, with multiple configurations achieving 61% on MedQA.

    “We see the highest performing debating strategy from those introduced in Section 2 is SoM with multiple different con-figurations achieving 61% on MedQA.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The distribution of total accuracy across datasets shows variability and robustness in each QA system's performance.

    “These plots demonstrate the variability and robustness of each system's performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    MAD typically requires a higher number of API calls, increasing the number of tokens to produce and process and ultimately the total running cost of the system.

    “MAD typically requires a higher number of API calls, increasing the number of tokens to produce and process and ultimately, the total running cost of the system.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Modulating the agreement in the prompt modifies the agreement on the first debate round, which in turn affects the outcome and final accuracy.

    “We can see that this parameter has a significant effect on the performance and that modulating the agreement in the prompt has the effect of modifying the agreement on the first debate round, which in turn affects the outcome and the final accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Society of Mind exhibits the largest surface area in the spider plot, suggesting it outperforms others across all debating metrics.

    “The initial plot compares the three systems, with Society of Mind (SoM) notably exhibiting the largest surface area, suggesting it outperforms others across all debating metrics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatEval exhibits a modest enhancement in relative accuracy from the debate process compared to the initial answer, Society of Mind shows a substantial increase, and Multi-Persona leads to a decrease.

    “Notably, the performance varies across the three systems: ChatEval exhibits a modest enhancement, Society of Mind shows a substantial increase, and Multi-Persona, intriguingly, leads to a decrease in performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    By tuning agreement intensity for Multi-Persona, it went from being the worst-performing protocol to the best-performing, beating out methods such as Medprompt and Self-Consistency.

    “By tuning this agreement intensity for Multi-Persona, it went from being the worst-performing protocol to the best-performing, beating out methods such as Medprompt and Self-Consistency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Modulating agreement intensity provides a substantial (≈15%) improvement in performance for Multi-Persona, and (≈5%) for SoM on the USMLE dataset.

    “We can see that modulating the agreement intensity in this way provides a substantial (≈15%) improvement in performance for Multi-Persona, and (≈5%) for SoM on the USMLE dataset.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    MedQA and PubMedQA directly benefit from a high agreement, while CIAR follows a reverse pattern.

    “Strikingly, the tendency is not always the same for each dataset: while MedQA and PubMedQA directly benefit from a high agreement, CIAR follows a reverse pattern.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    This facilitates multiple LLMs to participate in multi-agent and/or multi-round debates entirely using in-context prompting.

    “This facilitates multiple LLMs to participate in multi-agent and/or multi-round debates entirely using in-context prompting.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Several MAD strategies have been proposed to improve upon single-agent prompting methods, leading to improved performance on challenging natural language tasks.

    “Recently, several MAD strategies have been proposed to improve upon the enhanced reasoning capabilities of singleagent prompting methods leading to improved performance on challenging natural language tasks (Du et al., 2023;Liang et al., 2023;Chan et al., 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Du et al., 2023;Liang et al., 2023;Chan et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Beyond a certain threshold, additional computing does not guarantee better results, and performance is contingent on the right combination of hyperparameters and system design.

    “This suggests that, beyond a certain threshold, additional computing does not guarantee better results, and performance is both nuanced and contingent on the right combination of hyperparameters and system design.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    MAD protocols might not be inherently worse than other approaches, but they are more sensitive to different hyperparameter settings and difficult to optimize.

    “This suggests that MAD protocols might not be inherently worse than other approaches, but that they are more sensitive to different hyperparameter settings and difficult to optimize.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The source code for this work, including all MAD implementations and configurations, is publicly available at https://github.com/instadeepai/DebateLLM.

    “Source code for this work, including all MAD implementations and configurations, is publicly available at https://github.com/instadeepai/ DebateLLM.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The underperformance of MAD might primarily be due to MAD protocols being more sensitive to hyperparameters and not necessarily that debate inherently performs worse than single-agent methods.

    “Interestingly, we found that this might primarily be due to MAD protocols being more sensitive to hyperparameters and not necessarily that debate inherently performs worse than single-agent methods.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    MAD approaches could be as effective as, or even superior to, other strategies like Medprompt and Self-Consistency.

    “Our results show that MAD approaches could be as effective as, or even superior to, other strategies like Medprompt and Self-Consistency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Hyperparameter choices significantly influence system performance, with optimal settings tending to be dataset-specific.

    “These findings indicate that hyperparameter choices significantly influence system performance, with optimal settings tending to be dataset-specific.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    Larger areas in the spider plots are indicative of better performance.

    “This is further supported in the subsequent plots, which analyze the top three runs for each system and show that larger areas are indicative of better performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    Large-scale API-based benchmarking incurs substantial financial and time costs, which limit experiment scales and provide a barrier to replication and extension efforts.

    “Moreover, largescale API-based benchmarking incurs substantial financial and time costs, which both limit the experiment scales and provide a barrier of entry to replication and extension efforts.”
    Quote in the paperNot a substantial claim
24 of 24

  • The paper relies on this without support.

    There is no prior work comparing prompting strategies for multi-agent interactive reasoning, and no consensus for selecting one strategy over another.

    “However, to the best of our knowledge, there is no work comparing strategies, and there is no consensus for selecting one strategy over another.”

  • The authors state this as an assumption.

    MAD strategies might particularly shine on datasets requiring complex and commonsense reasoning.

    “The other datasets are difficult reasoning tasks, based on the assumptions that MAD strategies might particularly shine on datasets requiring complex and commonsense reasoning and be better able to escape reasoning traps.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Scaling the number of peer agents can yield diminishing, and sometimes negative, returns on complex repository-level coding tasks, partly because unconstrained communication introduces coordination overhead and failure modes of its own.

    “recent work suggests that scaling the number of peer agents can yield diminishing, and sometimes negative, returns on complex repository-level coding tasks, partly because unconstrained communication introduces coordination overhead and failure modes of its own (Smit et al., 2023;Kim et al., 2025).”

    From Adversarial Review: Structured Disagreement for Grounded Agentic Code Review (2026) · cites this paper as Smit et al., 2023

    The Experiments section reports that additional computing does not guarantee better results and that one debate setup reduced accuracy because a second agent contradicted correct answers. The Conclusions also note higher API and token costs for multi-agent debate. The paper evaluates question-answering and reasoning datasets, however, not repository-level coding tasks, and it does not establish unconstrained communication or coordination overhead as the cause. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • This design gives AR some benefits of role-separated cooperation without requiring a large round-table debate like MAD.

    “This design gives AR some benefits of role-separated cooperation without requiring a large round-table debate like MAD (Du et al., 2024;Smit et al., 2023;Wang et al., 2025).”

    From Adversarial Review: Structured Disagreement for Grounded Agentic Code Review (2026) · cites this paper as Smit et al., 2023

    The Multi-Agent Debate and Experiments sections describe agents with distinct roles and multi-round debate, while the Conclusions say MAD typically requires more API calls. But the paper does not establish that MAD requires a large round-table: its Multi-Persona testbed uses only two agents. It also does not assess AR's benefits. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • GPQA median exceeds the stated best Self-Consistency score

    Minor

    The table gives 0.24 as Self-Consistency's highest GPQA score but 0.29 as its median. A median cannot exceed the highest score in the same set.

  • Dataset categories add up to six, not seven

    Minor

    The stated three medical and three general datasets total six, although the paper says it evaluates seven. Chess is subsequently listed as a seventh dataset.

    Reported 7 · recomputed 6

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International conference on learning repre- sentations, 2021.
  2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.

Cited by

Sorted by citations

Related

Authors
Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2311.17371
arXiv
2311.17371
OpenAlex
W4389217455
Cited by
5
References
0
Citations by year
2026: 3 · 2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
multi-agent debate, large language models, honesty, prompting strategies, self-consistency, ensembling, hyperparameter optimization, accuracy-cost trade-off

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review