CoChat
Sign Up
PreprintCited by 32Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

Open access
Zhang, Hangfan, Cui, Zhiyao, Chen, Jianhao, Wang, Xinrun, Zhang, Qiaosheng, Wang, Zhen, Wu, Dinghao, Hu, Shuyue·2025·arXiv
DOI 10.48550/arxiv.2502.08788arXiv 2502.08788
Open this paper in CoChat — free accountOpen PDF

Abstract

Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs). Despite its conceptual appeal, current MAD research suffers from critical limitations in evaluation practices, including limited benchmark coverage, weak baseline comparisons, and inconsistent setups. This paper presents a systematic evaluation of 5 representative MAD methods across 9 benchmarks using 4 foundational models. Surprisingly, our findings reveal that MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation. To advance MAD research, we further explore the role of model heterogeneity and find it as a universal antidote to consistently improve current MAD frameworks. Based on our findings, we argue that the field must stop overvaluing MAD in its current form; for true advancement, we must critically rethink evaluation paradigms and actively embrace model heterogeneity as a core design principle.

AI snapshot

Written from the paper’s full text.

TL;DR

This position paper systematically evaluates five representative multi-agent debate (MAD) methods across nine benchmarks and four LLMs, finding that MAD generally fails to outperform simple single-agent baselines such as Chain-of-Thought (CoT) and Self-Consistency (SC). It argues that MAD evaluation must be rethought and that model heterogeneity—randomly drawing agents from different LLMs—consistently improves existing MAD frameworks.

Key findings

  • Across 36 configurations (4 models × 9 benchmarks), none of the evaluated MAD methods achieved a win rate higher than 20% against CoT; SoM, EoT, ChatEval, and AgentVerse outperformed CoT in approximately 15% of cases, while MP showed no significant improvement over CoT.
  • MAD underperformance versus SC was more pronounced: SC often achieved the highest performance when applicable, and MAD methods either showed no stable positive scaling with more tokens or continued to underperform SC at comparable token consumption.
  • Varying hyperparameters rarely reversed the negative result; the sole notable exception was EoT on GSM8K, where increasing agents from 3 to 9 led to continuous improvement and ultimately surpassed CoT.
  • Heter-MAD, which randomly selects from GPT-4o-mini and Llama3.1-70b with probability 0.5, consistently improved all considered MAD frameworks: Heter-SoM improved SoM-average by 6.4%, Heter-EoT improved EoT-average by 8.2%, and all considered MAD methods outperformed CoT-Average by up to 5.8%.

Methods

  • The authors evaluated five MAD frameworks (SoM, MP, EoT, AgentVerse, ChatEval) and three single-agent baselines (SA, CoT, SC) on nine benchmarks: MMLU, MMLU-Pro, AGIEval, CommonsenseQA, ARC-Challenge, GSM8K, MATH, HumanEval, and MBPP.
  • Foundation models were gpt-4o-mini-2024-07-18, claude-3-5-haiku-2024-1022, Llama3.1:8b-instruct, and Llama3.1:70b-instruct; inference used temperature T = 1 and top-p = 1, with default 6 LLM calls and three repeated runs.
  • For Heter-MAD, each agent queries a foundation model i with probability p_i from a candidate pool; validation used GPT-4o-mini and Llama3.1-70b with p = 0.5, instantiating SoM, EoT, ChatEval, and AgentVerse, while excluding MP.

Limitations

  • The paper states that it intentionally refrains from proposing technical solutions, does not aim to design a MAD framework optimized for model heterogeneity, and leaves such work to future research.
  • The paper states that its case studies are used for factual support rather than statistical analysis and acknowledges this may make the argument less convincing.
  • Evident from the text: the main evaluation covers four foundation models and nine benchmarks, Heter-MAD is tested with only two candidate models, MP is excluded from Heter-MAD, and the paper does not test optimized heterogeneous aggregation mechanisms.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.08788

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 4 cited sources checked · 1 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    EoT was shown to outperform CoT.

    “EoT was shown to outperform CoT (Yin et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Yin et al., 2023 source
  • Findingown data

    Increasing the number of agents or debate rounds does not significantly change outcomes in most scenarios.

    “Our empirical results, summarized in Figures 3a and 3b, indicate that in most scenarios, increasing the number of agents or debate rounds does not significantly change the outcomes.”
    Quote in the paperSubstantial claim
  • Findingown data

    MAD methods generally fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation.

    “Surprisingly, our findings reveal that MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation.”
    • Also stated in: § Win Tie Lose · plain
      “Our first set of experiments, comparing these MAD methods to single-agent approaches, including Chain-of-Thought (CoT) and Self-consistency (SC) (Wang et al., 2023), reveal an astonishing negative result-these MAD methods generally fail to outperform CoT, despite CoT being much simpler.”
    • Also stated in: § Does MAD outperform simple single-agent baselines? · hedged
      “Our results indicate that MAD methods fail to consistently outperform CoT across different models and benchmarks.”
    Quote in the paperSubstantial claim
  • Findingown data

    SoM always surpasses CoT on GSM8K when varying debate rounds.

    “Conversely, SoM always surpasses CoT on GSM8K when varying debate rounds.”
    Quote in the paperNot a substantial claim
  • Findingown data

    When SC can be applied, it achieves the highest performance in most cases, defeating CoT and MAD methods.

    “In most cases when SC can be appliedfoot_1 , SC achieves the highest performance, defeating CoT, not to mention MAD methods.”
    Quote in the paperSubstantial claim
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Premisecitation

    MP was evaluated against CoT solely on the CIAR dataset, which was initially proposed and has not been fully disclosed.

    “However, MP was evaluated against CoT solely on the CIAR dataset (Liang et al., 2024), which was initially proposed and has not been fully disclosed.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liang et al., 2024 source

      Table 3 compares MAD—not a method called MP—with CoT on counter-intuitive arithmetic reasoning. Appendix C (Table 10) also compares MAD with CoT on math and symbolic-reasoning datasets, contradicting “solely on the CIAR dataset.” The cited passages do not establish that the dataset was initially proposed there or has not been fully disclosed; page numbers are unavailable in our copy. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    Evaluations are frequently performed using only proprietary LLMs, and efficiency trade-offs of MAD have rarely been considered.

    “In addition, while evaluations are frequently performed using only proprietary LLMs, the efficiency of MAD that trade-off token consumption and performance gains have rarely been considered, with only few exceptions (Smit et al., 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Smit et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    SoM and EoT exhibit a more conservative approach, limiting frequency of errors while reducing ability to correct mistakes.

    “On the other hand, methods such as SoM and EoT exhibit a more conservative approach, effectively limiting the frequency of errors while also reducing their ability to correct mistakes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MP, ChatEval, and AgentVerse correct many wrong answers but also frequently introduce a high number of misstatements by altering initially correct answers.

    “We find that MP, ChatEval, and AgentVerse, while capable of correcting many wrong answers, also frequently introduce a high number of misstatements by mistakenly altering the initially correct answers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Aggregated over 36 configurations, SoM, EoT, ChatEval, and AgentVerse outperformed CoT in approximately 15% of cases, while MP did not demonstrate significant improvement over CoT.

    “As shown in Figure 2, SoM, EoT, ChatEval, and AgentVerse only outperformed CoT in approximately 15% cases, while MP did not demonstrate significant improvement over CoT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAD frameworks performed worse on programming tasks but better on mathematical reasoning relative to their overall performance.

    “When examining performance by task type, MAD frameworks performed worse on programming tasks (only SoM/AgentVerse had positive win rates) but better on mathematical reasoning, surpassing their overall performance levels.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Heter-MAD improves ChatEval average performance by +4.0% over CE-Average and +3.6% over CoT-Average.

    “ChatEval-Heter 82.6 ± 0.5 64.9 ± 0.4 78.8 ± 1.4 92.3 ± 0.5 70.5 ± 0.7 94.6 ± 0.2 71.4 ± 0.7 70.9 ± 0.8 49.8 ± 2.2 75.1 ± 0.3”
    • Also stated in: § Dataset MMLU MMLU-Pro CommensenseQA ARC-Challenge AGIEval GSM8K MATH HumanEval MBPP Average · plain
      “-vs CE-Average 3.6% 9.7% 2.4% 0.8% 2.5% 1.7% 11.5% 5.3% 1.4% +4.0% -vs CoT-Average 1.1% 11.3% -4.6% -1.2% 13.0% 1.9% 29.8% 0.8% -10.8% +3.6%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAD frameworks either show no positive trend in stable performance improvements with more inference budget or continue to underperform SC while consuming comparable tokens.

    “However, MAD frameworks either: (i) show no positive trend in achieving stable performance improvements with more inference budget, e.g., SoM does not stably achieve better performance on MMLU as consuming more tokens, or (ii) continue to underperform SC while consuming a comparable number of tokens although positively scaling up, e.g., EoT performs better as more tokens consumed on MMLU and GSM8K”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    EoT on GSM8K, increasing number of agents from 3 to 9 leads to continuous improvement, ultimately surpassing CoT.

    “The sole notable exception is observed with EoT on the GSM8K benchmark, where increasing the number of agents from 3 to 9 leads to a continuous improvement in performance, ultimately surpassing CoT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Heter-MAD improves EoT average performance by +8.2% over EoT-Average and +4.4% over CoT-Average.

    “EoT-Heter 79.7 ± 4.4 63.6 ± 3.1 83.9 ± 0.5 92.7 ± 0.5 69.8 ± 0.5 93.2 ± 0.3 73.1 ± 0.6 70.9 ± 1.1 54.3 ± 0.4 75.7 ± 0.6”
    • Also stated in: § Dataset MMLU MMLU-Pro CommensenseQA ARC-Challenge AGIEval GSM8K MATH HumanEval MBPP Average · plain
      “-vs EoT-Average 5.5% 14.7% 3.6% 3.3% 8.9% 8.4% 11.4% 9.7% 14.2% +8.2% -vs CoT-Average -2.5% 9.0% 1.5% -0.8% 11.8% 0.4% 32.9% 0.8% -2.8% +4.4%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Heter-MAD improves SoM average performance by +6.4% over SoM-Average and +5.8% over CoT-Average.

    “SoM-Heter 83.5 ± 0.1 65.0 ± 0.6 83.3 ± 0.1 92.1 ± 0.5 70.1 ± 0.4 94.6 ± 0.2 71.1 ± 0.9 75.8 ± 3.3 54.7 ± 1.9 76.7 ± 0.4”
    • Also stated in: § Dataset MMLU MMLU-Pro CommensenseQA ARC-Challenge AGIEval GSM8K MATH HumanEval MBPP Average · plain
      “-vs SoM-Average 4.8% 8.4% 2.5% 0.3% 10.8% 3.3% 7.2% 15.3% 11.3% +6.4% -vs CoT-Average 2.2% 11.4% 0.8% -1.4% 12.3% 1.9% 29.3% 7.8% -2.0% +5.8%”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Model heterogeneity is largely underexplored in MAD literature.

    “In this section, we investigate the potential of model heterogeneity in MAD, an aspect that has been largely underexplored in the literature.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Current MAD research typically instantiates multiple LLM agents from the same model.

  • May combine two claims.

    Findingown data

    Heter-MAD improves AgentVerse average performance by +2.7% over AGV-Average and +2.1% over CoT-Average.

    “-vs AGV-Average 2.1% 1.7% 0.9% -0.1% 2.1% 1.8% 5.8% 8.0% 5.2% +2.7% -vs CoT-Average 3.1% 8.0% -4.0% -0.9% 6.9% -2.3% 5.6% 11.6% -5.1% +2.1%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Datasets used for MAD evaluation are limited and have minimal overlap between different MAD methods.

    “The datasets used for evaluation are limited and have minimal overlap between different MAD methods”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The primary booster of Heter-MAD's performance gains is its ability to effectively handle CW and WC questions, which together account for a significant portion of the dataset.

    “The primary booster of Heter-MAD's performance gains is its ability to effectively handle CW and WC questions, which together account for a significant portion of the dataset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The CC category constitutes the largest proportion of questions.

    “Our observations reveal that the CC category constitutes the largest proportion of questions, and SoM-Heter consistently maintains high accuracy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Existing MAD approaches are less effective than currently believed, even underperforming simple single-agent methods like CoT and SC.

    “All these results point to a crucial reality-existing MAD approaches are less effective than currently believed, even underperforming simple single-agent methods like CoT and SC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    AgentVerse incorporates an additional execution-evaluation stage for programming tasks, which is usually absent in other MAD frameworks and arguably beyond the scope of MAD designs.

    “However, note that when handling programming tasks, AgentVerse incorporates an additional execution-evaluation stage so that agents can utilize the execution results from the generated programs, which is usually absent in other MAD frameworks and arguably beyond the scope of MAD designs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    There is a key trade-off in MAD design: overly aggressive methods may introduce instability, while conservative methods may struggle to capitalize on correction opportunities.

    “These observations not only explain why the MAD methods typically fail to outperform CoT and SC, but also highlight a key trade-off in the MAD design: overly aggressive methods may introduce instability, while conservative methods may struggle to capitalize on opportunities for correction.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    For fair comparison, the number of debate rounds was slightly adjusted to align all MAD methods to a similar inference budget measured by the number of LLM calls.

    “For fair comparison across different MAD methods, we slightly adjust the number of debate rounds of these methods to ensure that they all align to a similar amount of inference budget measured by the number of LLM calls.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    ChatEval and SoM were not compared to CoT in their papers, while EoT, AgentVerse, and MP were compared to CoT.

    “We acknowledge that ChatEval and SoM were not compared to CoT though, EoT, AgentVerse, and MP were compared to CoT in their papers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    This approach leads agents to emphasize the final answer, neglecting underlying reasoning steps.

    “This approach leads agents to emphasize the final answer, neglecting underlying reasoning steps.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    When responses diverge, rebuttals focus on outcome differences rather than analyzing the logic behind discrepancies.

    “When responses diverge, rebuttals focus on outcome differences rather than analyzing the logic behind discrepancies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Experiments were repeated three times, and standard deviations were reported.

    “For robustness, we repeated the experiments three times, reporting the standard deviations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Study design and conductnone given

    Determining whether a problem requires only a single knowledge point to solve, or whether the agent overly relies on the similarity or difference of answers, is challenging to achieve objectively.

    “Note that in these case studies, determining whether a problem requires only a single knowledge point to solve, or whether the agent overly relies on the similarity or difference of answers, is challenging to achieve objectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Most existing benchmarks predominantly include test cases that require only a single knowledge point for resolution, suggesting that more advanced single-agent methods could suffice and that MAD frameworks may be unnecessary in these contexts.

    “Most existing benchmarks predominantly include test cases that require only a single knowledge point for resolution, suggesting that more advanced single-agent methods could suffice and that MAD frameworks may be unnecessary in these contexts.”
    • Also stated in: § Limited benchmark · plain
      “Most test cases in existing benchmarks only require a single knowledge point to be solved, which naturally makes MAD unnecessary.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Current MAD methods lack fine-grained interaction capabilities, as agents engage in debates based solely on their complete responses to a given query.

    “Current MAD methods lack fine-grained interaction capabilities, as agents engage in debates based solely on their complete responses to a given query.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    For all MAD methods considered, the authors' open-source implementations were followed.

    “For all MAD methods considered,we follow the authors' open-source implementations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Heter-MAD and MAD-CoT improve MAD in distinct directions/perspectives.

    “Heter-MADandMAD-CoTimproveMAD in distinct directions.”
    • Also stated in: § D Additional Experimental Results · hedged
      “This observation suggests that Heter-MAD and MAD-CoT improve MAD from distinct perspectives”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Results from 36 experimental configurations were aggregated, and an ANOVA test with significance level 0.05 was conducted to categorize each comparison as Win, Tie, or Lose.

    “To gain a more rigorous and holistic view of MAD's performance relative to CoT, we aggregated results from 36 experimental configurations (four models, nine datasets).For each configuration, we conducted an ANOVA test with a significance level of 0.05 to assess whether MAD frameworks statistically outperformed, tied, or underperformed compared to CoT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

    Interpretationreasoning

    The significant performance gain of Heter-SoM and Heter-EoT stems from the rather simple design of SoM and EoT, and thus they are more compatible with heterogeneous models.

    “We hypothesize that the significant performance gain of Heter-SoM and Heter-EoT stems from the rather simple design of SoM and EoT, and thus they are more compatible with heterogeneous models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: SoM is the most simple and foundational MAD framework without complex mechanisms.
31 of 31

  • The paper relies on this without support.

    Current MAD research typically instantiates multiple LLM agents from the same model.

    “where, when multiple LLM agents are employed, they are typically instantiated from the same model.”

  • The authors state this as an assumption.

    Different LLMs trained on different data and paradigms may exhibit distinct strengths and weaknesses.

    “Intuitively, models trained on different data and paradigms may exhibit distinct strengths and weaknesses.”

  • The paper relies on this without support.

    Diversity of knowledge and reasoning among individuals enables meaningful discussion.

    “A key factor enabling meaningful discussion is the diversity of knowledge and reasoning among individuals.”

  • The paper relies on this without support.

    For proprietary LLMs, the number of tokens consumed is typically positively correlated with cost.

    “Moreover, for proprietary LLMs, the number of tokens consumed is typically positively correlated with cost.”

  • The paper relies on this without support.

    SoM is the most simple and foundational MAD framework without complex mechanisms.

    “Moreover, Heter-SoM, with SoM being the most simple and foundational MAD, achieves the highest performance, surpassing all the other, more recently developed MAD methods considered.”
    “as well as that SoM was the most foundational MAD framework without complex mechanisms.”

  • The paper relies on this without support.

    EoT, AgentVerse, and MP incorporate CoT-like mechanisms.

    “We note that while EoT, AgentVerse, and MP incorporate CoT-like mechanisms”
    “EoT, MP, and AgentVerse have incorporated CoT-style answers in their designs.”

  • The paper relies on this without support.

    SoM and ChatEval do not explicitly prompt agents to respond in a CoT style.

    “SoM and ChatEval do not explicitly prompt agents to respond in a CoT style.”

  • The authors state this as an assumption.

    The need for a single correct answer to be determined by majority voting.

    “We follow(Wang et al., 2023) which assumes the need for a single correct answer to be determined by majority voting.”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Llama-70B SoM MATH result differs between tables

    Major

    Table 4 reports 57.3 for SoM-Llama on MATH, but Table 7 reports 48.64 for SoM on Llama3.1-70B. This is substantially larger than a difference in printed precision.

  • Llama-70B EoT MATH result differs between tables

    Major

    Table 4 reports 55.3 for EoT-Llama on MATH, while Table 7 reports 43.63 for EoT on Llama3.1-70B. The remaining scores in the rows largely match.

  • GPT-4o-mini EoT GSM8K result differs between tables

    Minor

    Table 4 gives EoT-GPT a GSM8K score of 94.4, whereas Table 3 gives EoT on GPT-4o-mini a score of 91.40. The other scores in these rows largely agree.

  • SoM does not underperform CoT on all nine GPT-4o-mini datasets

    Minor

    Table 3 reports SoM above CoT on GSM8K (94.93 versus 93.60) and MATH (75.40 versus 72.87), and tied on MMLU-Pro (62.80 for both). The statement that it underperforms on all nine datasets contradicts those results.

  • Adding CoT does not consistently improve MAD

    Minor

    Table 9 reports ChatEval falling from 68.87 to 65.87 on AGIEval when CoT is added. SoM also falls from 94.93 to 94.67 on GSM8K, contradicting the unqualified claim of consistent improvement.

5 of 5

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I. and Zaremba, W. (2021). Evaluating large language models trained on code.
  2. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B. and Mordatch, I. (2023). Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning.
  3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. and Steinhardt, J. (2021b). Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) .
  4. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A. J., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A. M., La, A., Lampinen, A. K., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R. A., Lee, S. R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Debnath, S. S., Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S., Shieber, S., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V. V., vinay uday prabhu, Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, S., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z. and Wu, Z. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research Featured Certification.

Cited by

Sorted by citations

Related

Authors
Zhang, H., Cui, Z., Chen, J., Wang, X., Zhang, Q., Wang, Z., Wu, D., & Hu, S.
Affiliations
  • Pennsylvania State University (US)
  • Northwestern Polytechnical University (CN)
  • Nanjing University (CN)
  • Singapore Management University (SG)
  • Shanghai Artificial Intelligence Laboratory (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.08788
arXiv
2502.08788
OpenAlex
W4407571050
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multi-agent debate, large language models, evaluation benchmark, model heterogeneity, factual accuracy, chain-of-thought, self-consistency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review