CoChat
Sign Up
Cited by 302Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

Open access
Liang, Tian, He, Zhiwei, Jiao, Wenxiang, Wang, Xing, Wang, Yan, Wang, Rui, Yang, Yujiu, Shi, Shuming +1 more·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2024.emnlp-main.992
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks, which drives the research on cognitive behaviors of LLMs to explore human-like problem-solving strategies.Along this direction, one representative strategy is self-reflection, which asks an LLM to refine the solution with the feedback generated by itself iteratively.However, our study shows that such reflection-style methods suffer from the Degeneration-of-Thought (DoT) problem: once the LLM has established confidence in its solutions, it is unable to generate novel thoughts later through reflection even if its initial stance is incorrect.To address the DoT problem, we propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution.Clearly, our MAD framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation.Experiment results on two challenging datasets, commonsense machine translation and counterintuitive arithmetic reasoning, demonstrate the effectiveness of our MAD framework.Extensive analyses suggest that the adaptive break of debate and the modest level of "tit for tat" state are required for MAD to obtain good performance.Moreover, we find that LLMs might not be a fair judge if different LLMs are used for agents.Code is available at https://github. com/Skytliang/Multi-Agents-Debate.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper identifies a Degeneration-of-Thought (DoT) problem in LLM self-reflection, where models stick to incorrect initial answers, and proposes Multi-Agent Debate (MAD) with debaters and a judge to encourage divergent thinking. MAD improves over self-reflection and other baselines on commonsense machine translation and counter-intuitive arithmetic reasoning, though it adds inference cost and depends on debate design choices.

Key findings

  • On Common MT, GPT-3.5-Turbo with MAD surpasses GPT-4 on automatic and human metrics: lexical COMET 82.0 vs. 82.0, BLEURT 70.9 vs. 70.1, HUMAN 3.78 vs. 3.41; contextless COMET 84.8 vs. 84.7, BLEURT 73.7 vs. 73.6, HUMAN 3.67 vs. 3.63; contextual COMET 85.3 vs. 85.0, BLEURT 74.0 vs. 73.7, HUMAN 3.67 vs. 3.65.
  • On Counter-Intuitive AR, MAD with GPT-3.5-Turbo reaches 37.0% accuracy, outperforming GPT-3.5-Turbo (26.0), CoT (28.0), Self-Consistency (29.5), and Self-Reflect (27.5), but not GPT-4 (51.0).
  • MAD mitigates DoT relative to Self-Reflect: bias decreases from 29.0 to 24.8, and diversity increases from 19.3 to 49.7.
  • Analyses show adaptive break and a modest “tit for tat” level are needed for good performance; increasing debaters from 2 to 3 to 4 reduces Common MT performance (COMET 84.4 → 83.1 → 82.9; HUMAN 3.69 → 3.58 → 3.49), and LLM judges may favor the side sharing their backbone LLM.

Methods

  • MAD uses multiple debaters who argue in a fixed order under “tit for tat” meta prompts, plus a judge with discriminative mode to stop adaptively or extractive mode to produce a final answer; experiments set a maximum of 3 debate iterations.
  • Tasks: Commonsense Machine Translation (Chinese→English; 200 lexical, 450 contextless syntactic, 350 contextual syntactic examples) and Counter-Intuitive Arithmetic Reasoning (200 questions). Backbones include vicuna-7b-v1.5-16k, vicuna-13b-v1.5-16k, GPT-3.5-Turbo-0301, and GPT-4-0314; zero-shot with temperature 0.
  • Metrics: COMET, BLEURT, and human 1–5 scores for Common MT; accuracy for Counter-Intuitive AR. Baselines include Self-Reflect, Rerank, MAPS, CoT, and Self-Consistency.

Limitations

  • Stated: MAD requires more time cost because agents engage in multiple rounds; current LLM-based agents may struggle with coherence and relevance in long contexts; LLM-based judges may prefer outputs generated by themselves, so the paper recommends using the same LLM for all roles or distinct LLMs for judge and debaters.
  • Evident from the text: scaling to more debaters reduced performance, which the authors attribute to LLMs’ limitations in handling long texts and judges’ difficulty extracting information from longer debates.
  • Evident from the text: the main results are reported without variance, confidence intervals, or significance tests.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-main.992

Cites

0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 14 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    LLMs might not be a fair judge if different LLMs are used for agents.

    “we find that LLMs might not be a fair judge if different LLMs are used for agents.”
    • Also stated in: § 4.2 Analysis of Judge · hedged
      “This bias indicates that LLMs might not be a fair judge (Wang et al., 2023) when different LLMs are used for the agents.”
    • Also stated in: § 6 Conclusion · hedged
      “More interestingly, we find that LLMs might not be a fair judge if different LLMs are used for agents.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source
  • Interpretationcitation

    The key challenge of MAD with more debaters lies in the limitations of LLMs to handle long texts.

    “This suggests that the key challenge of MAD with more debaters lies in the limitations of the LLMs to handle long texts (Liu et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al., 2024 source
  • Findingown data

    In the Common MT case study, baseline GPT-3.5-Turbo and GPT-4 incorrectly translate the source words literally.

    “We can find that the baseline GPT-3.5-Turbo(even the more powerful GPT-4) incorrectly translates the source words literally.”
    Quote in the paperNot a substantial claim
  • Findingown data

    When Turbo debaters are involved, Vicuna serving as judge underperforms Turbo across all test sets.

    “When Turbo debaters are involved, Vicuna, serving as the judge, underperforms Turbo across all test sets.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Strong debaters with a weak judge work better than the reverse.

    “Strong debaters with a weak judge work better than the reverse.”
    Quote in the paperSubstantial claim
  • May combine two claims. 19
  • Classified as probably not a substantial claim. 20
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Modern large language models perform well on general language tasks but still struggle on complex reasoning tasks.

    “Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Zhu et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gou et al., 2023 source

      The Introduction describes remarkable progress across language tasks, and §4 evaluates models on mathematical and multi-hop reasoning. But the paper’s explicit finding about intricate reasoning concerns models’ difficulty verifying and correcting their own answers (Appendix D.1), not a general inability to perform complex reasoning tasks. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    DoT can result from bias and distorted perception, rigidity and resistance to change, and limited external feedback.

    “There are various factors (Bortolotti, 2011;Keestra, 2017) that could result in DoT, and we outline three here: (1) Bias and Distorted Perception.”
    • Also stated in: § 1 Introduction · hedged
      “(2) Rigidity and Resistance to Change.Self-reflection often involves challenging one's beliefs, assumptions, and behaviors.”
    • Also stated in: § 1 Introduction · hedged
      “(3) Limited External Feedback.Selfreflection is primarily an internal process, but external feedback can provide valuable perspectives and insights.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Bortolotti, 2011 source

      In “Decision-making and self-knowledge” (page not provided), Bortolotti says reflective deliberation may be manipulated by biased evidence and may overlook factors relevant to a choice, supporting the bias and distorted-perception factor. The passages do not identify rigidity, resistance to change, or limited external feedback as factors. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Keestra, 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Compared with MultiArith and GSM8K, the Counter-Intuitive AR dataset presents two distinct challenges: resistance to intuition and multi-step reasoning.

    “Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Roy and Roth, 2015

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Cobbe et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Concurrent studies also explore the multi-agent debate framework to enhance the reasoning ability of LLMs.

    “Concurrent with our work, a few studies (Xiong et al., 2023;Du et al., 2023) also explore the multi-agent debate framework to enhance the reasoning ability of LLMs.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Xiong et al., 2023 source
    • cites Du et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Addressing biases and stereotypes through self-reflection can be challenging.

    “In addition, addressing these biases and stereotypes through self-reflection can be challenging.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On Counter-Intuitive AR, MAD is not as good as GPT-4 but outperforms all other compared methods based on GPT-3.5-Turbo.

    “Our MAD framework, though not as good as GPT-4, outperforms all the other compared methods based on GPT-3.5-Turbo”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On Counter-Intuitive AR, Self-Reflect only marginally improves over baseline GPT-3.5-Turbo, while CoT and Self-Consistency bring more improvements.

    “Self-Reflect only marginally improves over the baseline GPT-3.5-Turbo, while CoT and Self-Consistency bring more improvements.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In forced 5-round debate or self-reflection, self-reflection shows low disagreement, suggesting that the LLM sticks to incorrect CoT answers and cannot engage in meaningful self-reflection.

    “The low disagreement of self-reflection suggests that the LLM sticks to the incorrect answers predicted by CoT and is unable to engage in meaningful selfreflection.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAPS and Self-Reflect achieve improvements over baseline GPT-3.5-Turbo on Common MT.

    “MAPS and Self-Reflec achieve improvements over baseline GPT-3.5-Turbo.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Complex questions require more iteration rounds of debate.

    “Complex questions require more iteration rounds of debate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The MAD framework effectively corrects inherent biases in translation, mitigates DoT, and considerably improves performance.

    “MAD framework effectively corrects inherent biases in translation, mitigates DoT, and considerably improves performance.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The questions in the Counter-Intuitive AR dataset are designed to stimulate counter-intuitive thinking, challenging conventional decision-making by presenting situations where the immediate, intuitive response is often incorrect.

    “The questions in our dataset are designed to stimulate counter-intuitive thinking, which aims to challenge conventional decision-making by presenting situations where the immediate, intuitive response is often incorrect.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    The Multi-Agent Debate (MAD) framework is effective on two challenging tasks, commonsense machine translation (Common MT) and counter-intuitive arithmetic reasoning (Counter-Intuitive AR).

    “We demonstrate the effectiveness of MAD on two challenging tasks and find that GPT-3.5-Turbo with MAD can even surpass GPT-4 on the Common MT dataset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Complex samples require more rounds of debate.

    “Complex samples require more rounds of debate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    An LLM-based judge may have a preference for outputs generated by itself.

    “LLM-based judge may have a preference for outputs generated by itself.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The proposed method requires more time cost, as agents need to engage in multiple rounds of interaction to present and refute arguments.

    “A limitation of this work is that our method requires more time cost, as agents need to engage in multiple rounds of interaction to present and refute arguments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The highest COMET score appears at the first iteration and is lower than the result of the adaptive break.

    “However, the highest COMET score appears at the first iteration and is also lower than the result of the adaptive break.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Each correct answer within the Counter-Intuitive AR dataset requires a rigorous multi-step reasoning process.

    “Each correct answer within the dataset requires a rigorous multi-step reasoning process, thereby evaluating the capacity of LLMs to engage in complex decision-making and problemsolving.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    LLMs made a common mistake initially in understanding the concept of average speed.

    “Here, LLMs made a common mistake initially in understanding the concept of average speed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The questions in the Counter-Intuitive AR dataset are embedded in hidden traps designed to elicit intuitive and appealing answers that are often incorrect.

    “The questions in our dataset are embedded in hidden traps designed to elicit intuitive and appealing answers that are often incorrect.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The MAD framework is less susceptible to the factors of DoT and can explore divergent chain-of-thoughts to achieve accurate answers.

    “Therefore, MAD is less susceptible to the factors of DoT, and can explore divergent chain-of-thoughts to achieve accurate answers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    In the challenging part of Common MT, authentic translation of each source sentence requires proper understanding of common sense knowledge.

    “Within the challenging part of Common MT, the authentic translation of each source sentence requires a proper understanding of common sense knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Common MT dataset is composed of Chinese-to-English translation examples used to examine lexical and contextless/contextual syntactic ambiguity resolution abilities.

    “The Common MT dataset is composed of Chinese⇒English translation examples (He et al., 2020), which are used to examine three types of ambiguity resolution abilities of translation models, covering lexical and contextless/contextual syntactic ambiguity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites He et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Self-reflection can be effective in creating better answers but is highly dependent on the self-evaluation capabilities of LLMs, which are not formally guaranteed.

    “While self-reflection can be effective in creating better answers, it is highly dependent on the self-evaluation capabilities of LLMs, which are not formally guaranteed (Shinn et al., 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Shinn et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Human evaluation randomly shuffled the display order of translated sentences from all methods in an anonymous manner.

    “We randomly shuffled the display order of the translated sentences from all methods in an anonymous manner.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Within the challenging part of Common MT, the authentic translation of each source sentence requires a proper understanding of common sense knowledge.

    “Within the challenging part of Common MT, the authentic translation of each source sentence requires a proper understanding of common sense knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    For bias evaluation, annotators labeled each sentence as 1 (not conforming to commonsense) or 0 (conforming to commonsense), and reported the degree of bias for the whole test set.

    “Here, we asked the annotators to label each sentence as 1 (not conforming to commonsense) or 0 (conforming to commonsense), and report the degree of bias for the whole test set.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The MAD method helps LLMs understand semantics correctly and generate highly authentic translations.

    “In response to this issue, our MAD method helps LLMs understand semantics correctly and generate highly authentic translations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Human evaluation used three professional human translators with Krippendorff's Alpha = 0.76 to directly assess all methods together.

    “Then, employed three professional human translators (Krippendorff's Alpha = 0.76) to directly assess all methods together.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Failure to identify and address such ambiguities may result in inaccurate translations.

    “Failure to identify and address such ambiguities may result in inaccurate translations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The MAD method encourages divergent thinking and effectively prevents intuitive but incorrect responses, leading to the correct method for calculating average speed.

    “Our MAD method encourages divergent thinking and effectively prevents intuitive but incorrect responses, leading to the correct method for calculating average speed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Common MT dataset is composed of Chinese-to-English translation examples and is used to examine three types of ambiguity resolution abilities of translation models.

    “The Common MT dataset is composed of Chinese⇒English translation examples (He et al., 2020), which are used to examine three types of ambiguity resolution abilities of translation models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites He et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Previous studies on thinking hierarchy suggest that humans have a fast and intuitive system and a slow and logical system, and tend to run the lower level system before the higher level one.

    “Previous studies on thinking hierarchy (Daniel, 2017) suggest that we humans have a fast and intuitive system and a slow and logical system, and tend to run the lower level system before the higher level one.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Daniel, 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Forcing the debate to continue harms translation results, which demonstrates the reasonableness of the adaptive break strategy.

    “Forcing the debate to continue will harm the translation results, which demonstrates the reasonableness of our adaptive break strategy.”
    Quote in the paperSubstantial claim
34 of 34

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MP was evaluated against CoT solely on the CIAR dataset, which was initially proposed and has not been fully disclosed.

    “However, MP was evaluated against CoT solely on the CIAR dataset (Liang et al., 2024), which was initially proposed and has not been fully disclosed.”

    From Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity (2025) · cites this paper as Liang et al., 2024

    Table 3 compares MAD—not a method called MP—with CoT on counter-intuitive arithmetic reasoning. Appendix C (Table 10) also compares MAD with CoT on math and symbolic-reasoning datasets, contradicting “solely on the CIAR dataset.” The cited passages do not establish that the dataset was initially proposed there or has not been fully disclosed; page numbers are unavailable in our copy. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The idea of independent answer generation has been previously explored in different multi-agent settings, but these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.

    “While the idea of independent answer generation has been previously explored in different multi-agent settings (Du et al., 2023b;Liang et al., 2024), these studies did not systematically formalize and evaluate it separately from other multi-agent discussion parameters.”

    From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Liang et al., 2024

    The Multi-Agent Debate Framework describes debaters speaking in a fixed order, with each response based on the previous debate history; the negative debater is instructed to disagree with the affirmative debater’s points. That is contrary to citing this paper as an example of independent answer generation. The paper analyzes debate features such as disagreement and adaptive stopping, but does not establish the claimed independent-generation step. (LLM, conf=0.83)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Each round of dialogue in the collaborative process refines artifacts from different aspects, naturally elevating the probability of producing more nuanced artifacts.

    “Consequently, each round of dialogue in the collaborative process refines artifacts from different aspects, naturally elevating the probability of producing more nuanced artifacts (Liang et al., 2024; Du et al., 2024a; Cohen et al., 2023).”

    From Scaling Large Language Model-based Multi-Agent Collaboration (2024) · cites this paper as Liang et al., 2024

    The translation example in F.1 shows debaters considering different interpretations and reaching a more nuanced translation. But Section 4.3 says the best translation is often reached after one round and that forcing further debate can harm results. The paper therefore does not support the claim that each round naturally improves the artifact. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Multi-agent collaboration has proven beneficial in uniting the expertise of diverse agents for autonomous task-solving.

    “Along this line, multi-agent collaboration has proven beneficial in uniting the expertise of diverse agents for autonomous task-solving”

    From Scaling Large Language Model-based Multi-Agent Collaboration (2024) · cites this paper as Liang et al., 2024

    The Introduction reports that agents exchange arguments and feedback to reach more accurate answers, and that MAD outperforms baselines on two reasoning tasks. But the Generative Agents section says enhanced performance can come from agents using the identical backbone LLM; the paper does not establish that the benefit comes from uniting agents’ diverse expertise or demonstrate autonomous task-solving broadly. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • MAD has been proposed as a way to improve the reasoning and factuality of LLM outputs by allowing multiple agents to exchange and revise their answers.

    “In parallel, Multi-Agent Debate (MAD) has been proposed as a way to improve the reasoning and factuality of LLM outputs by allowing multiple agents to exchange and revise their answers (Du et al., 2024; Liang et al., 2024).”

    From Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation (2026) · cites this paper as Liang et al., 2024

    The Introduction says MAD lets agents correct one another and achieve more accurate answers, and Section 2 describes debaters exchanging arguments across iterations before a judge selects a solution. The paper reports improvements on translation and arithmetic-reasoning tasks, but does not establish improved factuality of LLM outputs more broadly. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few- shot learning for machine translation. In Inter- national Conference on Machine Learning, pages 10867-10878.
  2. PMLR.
  3. Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shum- ing Shi, and Xing Wang. 2024. Exploring human- like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229-246.

Cited by

Sorted by citations

Related

Authors
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z.
Affiliations
  • Shanghai Jiao Tong University (CN)
  • Tencent (China) (CN)
  • Tsinghua University (CN)
ORCID
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2024
Pages
17889-17904
DOI
10.18653/v1/2024.emnlp-main.992
OpenAlex
W4404782209
Cited by
274
References
0
Citations by year
2026: 136 · 2025: 121 · 2024: 18 · 2023: 1
FWCI
87.44
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, multi-agent debate, divergent thinking, self-reflection, tit-for-tat, LLM-as-a-judge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review