Diversity of Thought Elicits Stronger Reasoning Capabilities in Multi-Agent Debate Frameworks

May combine two claims.

The 7B diverse framework gained 17% by the fourth round and the 2B framework gained 10%.

“the 7B model framework achieved a significant performance increase of 17% by the 4th round, while the 2B model framework saw a 10% improvement”

May combine two claims. The quote only partly states this claim.

Tested individually on GSM-8K, Gemini-Pro, PaLM 2-M and Mixtral 7B×8 score 78%, 64% and 70% respectively.

“When tested individually, these models achieved accuracy rates of 78%, 64%, and 70%, respectively, on the benchmark.”

May combine two claims.

The diverse medium-capacity model set reaches 94% on ASDiv after four debate rounds, a new state of the art on that benchmark.

“this diverse set of medium-capacity models sets a new state-of-the-art performance on the ASDiv benchmark (94%)”

• Also stated in: § 4.2. Diversity of Thought · plain

  “After 4 rounds of debate, the framework reached an accuracy of 94%, setting a new state-of-the-art performance on the ASDiv benchmark”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Tested individually on ASDiv, Gemini Flash 1.5, Gemini Pro and GPT 3.5 score 89%, 86% and 81% respectively.

“Gemini Flash 1.5, Gemini Pro, and GPT 3.5 achieving accuracy rates of 89%, 86%, and 81%, respectively”

May combine two claims.

By the fourth debate round the diverse framework exceeds GPT-4 and Gemini Ultra on MATH by 24 and 14 percentage points respectively.

“the framework outperformed both GPT-4 and Gemini Ultra by substantial margins of 24% and 14%, respectively”

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

After four rounds of debate, a diverse set of medium-capacity models (Gemini-Pro, Mixtral 7B×8, PaLM 2-M) reaches 91% accuracy on GSM-8K, outperforming GPT-4.

“after 4 rounds of debate, a diverse set of medium-capacity models (Gemini-Pro, Mixtral 7B×8, and PaLM 2-M) outperforms GPT-4 on the GSM-8K benchmark, scoring 91% accuracy”

• Also stated in: § 4.2. Diversity of Thought · plain

  “the accuracy of the framework improved significantly, rising from 78% to an impressive 91% after 4 rounds of debate”

May combine two claims.

Tested individually on MATH, Gemini Flash 1.5, Gemini Pro and GPT 3.5 score 55%, 32% and 33% respectively.

“the individual models achieved accuracy rates of 55%, 32%, and 33% for Gemini Flash 1.5, Gemini Pro, and GPT 3.5, respectively”

May combine two claims.

A homogeneous debate of three Gemini-Pro instances reaches only 82% on GSM-8K (80% without chain-of-thought, 82% with zero-shot chain-of-thought).

“when 3 instances of Gemini-Pro are used, performance only reaches 82%”

• Also stated in: § 4.2. Diversity of Thought · plain

  “the performance improved from 78% to 80% without chain-of-thought (COT) reasoning and to 82% with zero-shot COT”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the illustrative debate example, Mixtral kept its original answer for the first two rounds and began to shift in the third round after weighing the other models' reasoning.

“Mixtral, one of the participating models, initially maintained its original answer for the first two rounds of the debate.”

• Also stated in: § 4.2. Diversity of Thought · plain

  “However, by the third round, Mixtral's stance began to shift as it carefully considered the reasoning put forth by the other two models.”

May combine two claims.

The experimental setup was held constant across evaluations, with identical starting prompts and language models used for the baseline and the debate framework.

“we maintain a consistent experimental setup across all evaluations.We use identical starting prompts and language models for both the baseline and the multi-agent debate framework approaches.”

May combine two claims.

What an effective debate requires is diverse model architectures of similar capacity, rather than prior competence on the benchmark.

“the critical requirement for an effective debate is the presence of diverse model architectures of similar capacity, which induces learning and enhances reasoning capabilities”

• AssumptionRests on an assumption: Using models drawn from different families and architectures of comparable capacity amounts to the intended 'diversity of thought'.

Classified as probably not a substantial claim.

Minsky's Society of Mind theory holds that agents differing in method, purpose, knowledge representation and result production can improve factual accuracy through debate and communication.

“The multi-agent debate approach, inspired by Minsky [1988]'s "Society of Mind" theory, posits that diverse agents approaching a problem with different methods, purposes, knowledge representations, and result-production techniques can enhance factual accuracy through debate and communication.”

• cites Minsky [1988]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Chain-of-thought reasoning is an emergent ability that appears as model size increases.

“chain-of-thought (COT) reasoning is an emergent ability that arises with increased model size”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Even smaller-scale models can benefit from the collaborative reasoning of debate, so the approach can be used in resource-constrained settings.

“even smaller-scale models can benefit from the collaborative reasoning process facilitated by the debate framework, without the need for resource-intensive large-scale models”

Probably not a substantial claim: may not pass S3 (load-bearing).

Model prompts were configured to end each response with a boxed answer, giving a standardized format for evaluation.

“We configured the model prompts to consistently provide a boxed answer at the conclusion of each response, ensuring a standardized format for evaluation.”