CoChat
Sign Up
PreprintCited by 0Topic ModelingMathematics Education and PedagogyAI-based Problem Solving and Planning

Critic-Guided Heterogeneous Multi-Agent Reasoning for Reliable Mathematical Problem Solving

Open access
Sharif, Muhammad Talha, Rehman, Abdul·2026·arXiv
DOI 10.48550/arxiv.2606.05704arXiv 2606.05704
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent Large Language Models (LLMs) have shown impressive reasoning abilities; but they are still susceptible to hallucinations, intermediate reasoning mistakes, and unreliable reasoning results in complex mathematical reasoning problems. In this study, we introduce a critic-based heterogeneous multi-agent approach to improve the dependability of mathematical reasoning. This framework incorporates several LLM agents of different specialties and employs a critic-driven adaptive learning system to assess and guide the reasoning process based on intermediate feedback. The system adopts a generator-validator framework, with the validator not only determining correctness but also offering critiques to guide regeneration of solutions. This allows for adaptive error correction and prevents error cascading. Our experiments on the GSM8K benchmark show that the proposed method achieves up to 13% accuracy improvement over single-shot and non-critic models. Additionally, findings suggest that heterogeneity and critique reduce the need for large models, allowing smaller models to perform on par. Ablation studies reveal the main performance gains are due to the critic-based feedback loop and not model size. In summary, the proposed approach showcases the benefits of combining heterogeneous multi-agent collaboration and critique to obtain reliable and interpretable reasoning systems.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a critic-guided heterogeneous multi-agent generator-validator framework for mathematical word problems, where a validator critiques failed solutions and triggers regeneration. On the full GSM8K test set, critique-guided setups reach 85.44–93.56% accuracy versus 72.55–80.14% without critique, with the best configuration at 93.56%.

Key findings

  • Without critic, accuracy ranges from 72.55% to 80.14% (Experiments 2A, 2B, 1A, 1B); with critic, accuracy rises to 85.44–93.56% (3A, 3B, 4A, 4B), showing up to approximately 13% improvement.
  • The best result is 93.56% (Experiment 4B: llama-3.1-8b-instant generator, openai/gpt-oss-20b validator, critique + multiround), outperforming RDoLT’s 90.98% with ChatGPT-4o by +2.58% absolute.
  • Increasing validator size yields smaller gains than critique: +6.30% without critic and +4.93% with critic, compared with critic-based improvement of +12–16%. With critique, smaller validators perform on par or better than larger ones.
  • Retry logs show most problems solved on first try (73–78%), with recovered samples of +161 (3A), +210 (3B), +193 (4A), and +205 (4B), equivalent to 12–16% improvement due to critique; failures drop to e.g. 85 in 4B.

Methods

  • Uses the full GSM8K test set (1,319 examples) from openai/gsm8k, configuration main, with gold answers extracted from reference solutions; no fine-tuning or additional training, only inference.
  • Two-step multi-agent model: a fixed generator (llama-3.1-8b-instant) produces concise solutions with 3–4 reasoning steps and a final answer; a validator judges reasoning-step correctness and final-answer correctness, returning JSON flags and sometimes a critique.
  • Eight experimental setups compare single-shot vs. critique + multiround validation, homogeneous vs. heterogeneous validators, and small vs. large validator models (8B, 20B, 70B, 120B); if first solution fails validation, validator critique guides regeneration, with accuracy as the main metric.

Limitations

  • The paper does not state a dedicated limitations section.
  • Evident from the text: evaluation is limited to GSM8K (1,319 examples) and a single generator model (llama-3.1-8b-instant), so generalization to other math datasets or generators is not tested.
  • Evident from the text: the method depends on validator critiques and API-based inference; diagnostic metrics such as validator parse errors are mentioned but not reported in the results.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2606.05704

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 5 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The number of failures is dramatically decreased (e.g., 85 in 4B).

    “The number of failures is also dramatically decreased (e.g., 85 in 4B).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
  • Findingown data

    The proposed critic-guided heterogeneous multi-agent method achieves up to 13% accuracy improvement over single-shot and non-critic models on the GSM8K benchmark.

    “Our experiments on the GSM8K benchmark show that the proposed method achieves up to 13% accuracy improvement over single-shot and non-critic models.”
    • Also stated in: § V. RESULTS AND DISCUSSIONS · hedged
      “With critic, accuracy boosts to 85.44% -93.56% (3A, 3B, 4A, 4B), showing up to approximately 13% improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
    • Rests on an assumption: The accuracy metric (fraction where reasoning consistency and final answer are correct) is a valid measure of mathematical reasoning performance.
  • Findingown data

    Experiments 1A (70B validator) and 1B (120B validator) have almost identical performance (80.14% vs 80.06%).

    “For instance, 1A (70B: 80.14%) and 1B (120B: 80.06%) have almost identical performance, suggesting limited gains from single validation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
  • Findingown data

    Ablation studies reveal that the main performance gains are due to the critic-based feedback loop and not model size.

    “Ablation studies reveal the main performance gains are due to the critic-based feedback loop and not model size.”
    • Also stated in: § V. RESULTS AND DISCUSSIONS · plain
      “Comparing singleshot (1A-2B) with critic experiments (3A-4B) showing that the performance improvements are mostly driven by the use of critique-based feedback.”
    • Also stated in: § VI. CONCLUSION · plain
      “This comparison of single-shot and critique-guided experiments is an ablation study, which shows that the performance improvement is mainly due to the introduction of critique rather than model scaling.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
  • Findingown data

    Most problems are solved on first try (73-78%).

    “Most problems are solved on first try (73-78%), but retries are important as 3A: +161 recovered, 3B: +210 recovered, 4A: +193 recovered, 4B: +205 recovered.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
  • May combine two claims. 13
  • Classified as probably not a substantial claim. 6
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Larger validators provide little to no improvement without critique, while the critique mechanism consistently improves performance across all configurations.

    “Our results show that larger validators provide little to no improvement without critique, while the critique mechanism consistently improves performance across all configurations.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Findingown data

    The proposed critic-guided system achieves 93.56% accuracy on GSM8K, outperforming the best reported result from the RDoLT framework (90.98% with ChatGPT-4o) by +2.58% absolute.

    “The proposed critic-guided system achieves 93.56% accuracy on GSM8K, outperforming the best reported result from the RDoLT framework [13] (90.98% with ChatGPT-4o) by +2.58% absolute, demonstrating the effectiveness of critiqueguided multi-agent reasoning.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
    • Rests on an assumption: The accuracy metric (fraction where reasoning consistency and final answer are correct) is a valid measure of mathematical reasoning performance.

  • May combine two claims.

    Findingown data

    Without critic, larger validators are beneficial in homogeneous setups (+7.59%) but not in heterogeneous setups (+1.21%).

    “Without critic, larger validators are beneficial, particularly in homogeneous (+7.59%) but not in heterogeneous (+1.21%) setups.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Findingown data

    The best configurations obtain accuracy of 72.55%-80.14% for direct generation and 85.44%-93.56% for critique-guided generation, with highest 93.56%.

    “The best configurations obtain an accuracy of 72.55% -80.14% for direct generation and 85.44% -93.56% for critique-guided generation, with the highest accuracy of 93.56%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.
    • Rests on an assumption: The accuracy metric (fraction where reasoning consistency and final answer are correct) is a valid measure of mathematical reasoning performance.

  • May combine two claims.

    Findingown data

    Increasing validator size (8B to 20B) yields +6.30% (no critic) and +4.93% (with critic).

    “Increasing validator size (8B to 20B) yields, +6.30% (no critic) and +4.93% (with critic).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Findingown data

    The critic-based improvement (+12-16%) is much larger than the improvement from increasing validator size (+6.30% no critic, +4.93% with critic).

    “This is much smaller than the critic-based improvement of +12-16%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Findingown data

    Critic-guided setups perform better: 3A (85.44%), 3B (92.04%), 4A (88.63%), and 4B (93.56%).

    “Critic-guided setups perform better: 3A (85.44%), 3B (92.04%), 4A (88.63%), and 4B (93.56%), indicating the critique loop is the key to reliability.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Findingown data

    Retries recover additional problems: 3A +161, 3B +210, 4A +193, 4B +205.

    “Most problems are solved on first try (73-78%), but retries are important as 3A: +161 recovered, 3B: +210 recovered, 4A: +193 recovered, 4B: +205 recovered.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The same generator model, dataset, prompts and generator sampling settings are used in all experiments.

    “We use the same generator model, dataset, prompts and generator sampling settings in all experiments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The approach offers a scalable and inference-only approach for improving accuracy and robustness without further training.

    “Our approach offers a scalable and inference-only approach for improving accuracy and robustness without further training.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    In diverse settings, the difference between large and small validators is negligible with critique, suggesting adaptive reasoning processes mitigate the need for larger models.

    “In particular, in diverse settings, the difference between large and small validators is negligible with critique, suggesting that adaptive reasoning processes mitigate the need for larger models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • The quote only partly states this claim.

    Interpretationown data

    Many errors are amendable.

    “Finally, our analysis of multi-round validation reveals that 12%-16% of initially incorrect solutions are corrected during iterative problem solving, suggesting that many errors are amendable.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Interpretationown data

    Heterogeneity and critique reduce the need for large models, allowing smaller models to perform on par.

    “Additionally, findings suggest that heterogeneity and critique reduce the need for large models, allowing smaller models to perform on par.”
    • Also stated in: § V. RESULTS AND DISCUSSIONS · boosted
      “This demonstrates that heterogeneity and critique lower the need for large models.”
    • Also stated in: § VI. CONCLUSION · hedged
      “In particular, in diverse settings, the difference between large and small validators is negligible with critique, suggesting that adaptive reasoning processes mitigate the need for larger models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The validator model's judgments of reasoning validity and final answer correctness are accurate.

  • May combine two claims.

    Interpretationreasoning

    The system allows for adaptive error correction and prevents error cascading.

    “This allows for adaptive error correction and prevents error cascading.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A heterogeneous multi-agent model where agents are powered by different LLMs can be very effective because of complementary characteristics of the models.

    “However, the authors show that a heterogeneous multi-agent model where agents are powered by different LLMs could be very effective because of the complementary characteristics of the models.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    RDoLT substantially outperforms existing approaches such as Chain-of-Thought and ReAct on GSM8K, with high accuracy and greater stability.

    “The results of experiments on data sets such as GSM8K show that RDoLT substantially outperforms existing approaches such as Chain-of-Thought and ReAct, with high accuracy and greater stability in terms of variance in the reasoning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The system uses no fine-tuning or additional training; only inference is used.

    “No fine-tuning or additional training is involved; only inference is used.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All experiments are assessed on the entire GSM8K test set with identical generation and validation settings.

    “We assess all experiments on the entire GSM8K test set with identical generation and validation settings.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Multi-agent debate is often unable to outperform powerful single-agent algorithms such as chain-of-thought prompting or self-consistency.

    “The paper indicates that the multi-agent debate is often unable to outperform powerful single-agent algorithms such as chain-of-thought prompting or self-consistency.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

19 of 19

  • The paper relies on this without support.

    LLMs are susceptible to hallucinations, intermediate reasoning mistakes, and unreliable reasoning results in complex mathematical reasoning problems.

    “Recent Large Language Models (LLMs) have shown impressive reasoning abilities; but they are still susceptible to hallucinations, intermediate reasoning mistakes, and unreliable reasoning results in complex mathematical reasoning problems.”
    “Common issues include hallucinations, coherence of chains-of-thought and propagation of errors, where a mistake in the middle of a solution process is continued until the end of the chain.”

  • The paper relies on this without support.

    Existing multi-agent systems (e.g., debate-based or sequential collaboration) are not sufficient to detect and correct intermediate reasoning errors.

    “The existing multi-agent systems e.g.debate-based or sequential collaboration pipeline, increase the reasoning diversity, but are not sufficient to detect and correct intermediate reasoning errors.”

  • The paper relies on this without support.

    Most existing multi-agent systems are homogeneous or lack an adaptive model to assess the reasoning.

    “But many of the existing systems either consist of homogeneous agents or don't have an adaptive model to assess the reasoning.”

  • The paper relies on this without support.

    The validator model's judgments of reasoning validity and final answer correctness are accurate.

    “The validator is different for each experiment and judges the correctness of the reasoning steps and the final answer.”

  • The paper relies on this without support.

    The accuracy metric (fraction where reasoning consistency and final answer are correct) is a valid measure of mathematical reasoning performance.

    “We measure performance with the metric of accuracy, which is the fraction of examples where reasoning consistency and the final answer are correct.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Method specifies two rounds but retry log reports a third

    Minor

    The method says correctness is determined across either of two rounds, while Table III records third-round passes for three experiments.

  • Table IV assigns experiment 3B the accuracy of 4B

    Minor

    Table IV labels the 93.56% result as experiment 3b, whereas the results identify 3B as 92.04% and 4B as 93.56%.

  • Recovery percentage uses all problems, not initially incorrect ones

    Minor

    The stated 12%-16% is approximately the number recovered divided by all 1,319 problems, not by initially incorrect solutions. For example, 3A initially fails 1,319 − 966 = 353 problems and recovers 161 of them, or about 45.6%.

3 of 3

Cites

Flagged first, then by citations

Authors
Sharif, M. T., & Rehman, A.
Affiliations
National University of Computer and Emerging Sciences (PK)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2606.05704
arXiv
2606.05704
OpenAlex
W7163677127
Semantic Scholar
b9bf25dffb058bdad00951412a29f1224c97f50b
Cited by
0
References
15
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Mathematics › Applied Mathematics › Mathematics Education and Pedagogy
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
mathematical reasoning, large language models, heterogeneous multi-agent systems, hallucination mitigation, error correction, GSM8K, reasoning reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review