CoChat
Sign Up
Cited by 4Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning

Open access
Chen, Justin, Prasad, Archiki, Saha, Swarnadeep, Stengel-Eskin, Elias, Bansal, Mohit·2025·Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2025.emnlp-main.1660
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language model (LLM) reasoning can be improved by scaling test-time compute with aggregation, i.e., generating multiple samples and aggregating over them.While improving performance, this strategy often reaches a saturation point beyond which additional compute provides no return.Refinement offers an alternative by using model-generated feedback to improve answer quality.However, refinement faces three key challenges: (1) Excessive refinement: Uniformly refining all instances can cause over-correction and reduce overall performance.(2) Inability to localize and address errors: LLMs struggle to identify and correct their own mistakes.(3) Insufficient refinement: Stopping refinement too soon could leave errors unaddressed.To tackle these issues, we propose MAGICORE, a framework for Multi-Agent Iteration for Coarse-to-fine Refinement.MAGICORE mitigates excessive refinement by categorizing problems as easy or hard, solving easy problems with coarsegrained aggregation, and solving the hard ones with fine-grained multi-agent refinement.To better localize errors, we incorporate external step-wise reward model scores, and to ensure sufficient refinement, we iteratively refine the solutions using a multi-agent setup.We evaluate MAGICORE on Llama-3-8B and GPT-3.5 and show its effectiveness across seven reasoning datasets.One iteration of MAGI-CORE beats Self-Consistency by 3.4%, Bestof-k by 3.2%, and Self-Refine by 4.0% even when these baselines use k = 120, and MAGI-CORE uses less than 50% of the compute. 1

AI snapshot

Written from the paper’s full text.

TL;DR

MAGICORE is a multi-agent, iterative, coarse-to-fine refinement framework that classifies problems as easy or hard using reward-model scores and answer entropy, then applies weighted self-consistency to easy problems and multi-agent review-refine to hard ones. It improves over aggregation and refinement baselines across seven reasoning datasets with Llama-3-8B and GPT-3.5 while using less compute.

Key findings

  • One iteration of MAGICORE beats Self-Consistency by 3.4%, Best-of-k by 3.2%, and Self-Refine by 4.0% even when baselines use k = 120, while using less than 50% of the compute. On Llama-3-8B, MAGICORE Iter=3 reaches 75.6% average accuracy versus 71.0% for Best-of-k (k = 120), 70.8% for k-way SC (k = 120), and 70.2% for Self-Refine + k-way SC; on GPT-3.5, it reaches 80.9% versus 77.3% for Best-of-k.
  • Selective refinement avoids over-correction: uniformly refining all samples drops performance by 3.8% on MMLU and 5.2% on MATH relative to aggregation-only, while MAGICORE’s selective refinement improves up to 2.6% over uniformly applying aggregation. On MATH, MAGICORE improves 6.4% over the strongest Best-of-k baseline, whereas uniform refinement can drop 5.2%.
  • Iterative refinement helps: MAGICORE stabilizes at the third iteration with 75.6% average accuracy, a 1.4% improvement over its first iteration, while Self-Refine + SC fluctuates around 70%. It also generalizes to ARC-Challenge and Date Understanding, reaching 88.5 and 80.5 at Iter=3 versus 86.0 and 72.5 for 120-way SC.
  • Ablations show all three issues must be addressed jointly: addressing only Issue 1 gives 64.7 MMLU / 44.0 MATH, only Issue 2 gives 65.9 / 45.4, only Issue 3 gives 60.3 / 36.4, while MAGICORE gives 68.9 / 47.8. PRM step scores for refinement outperform LLM self-refinement, random step scores, and ORM-only scores.

Methods

  • Base LLMs are Llama3-8B-Instruct and GPT-3.5-Turbo; ORM is InternLM-7B and PRM is Math-Shepherd-7B. Default sampling is k = 40 reasoning chains per iteration, temperature 0.8, and maximum 3 iterations. Evaluation uses seven datasets: MMLU-Math (974), MATH (500 subset), SVAMP (1,000), GSM8K (1,312), SAT (220), ARC-Challenge (200), and Date Understanding (200).
  • Difficulty classification: generate k solutions, cluster by final answer, and check two conditions. Condition 1 tests whether the majority answer’s normalized average RM score is ≥ 0. Condition 2 tests RM answer confidence via entropy of the RM-weighted answer distribution, mapped through a sigmoid with threshold 0.5. If either condition is true, the instance is easy and uses Weighted Self-Consistency; if both fail, it is hard and uses fine-grained refinement.
  • Multi-agent refinement: the Solver generates k chains; the Reviewer uses step-wise PRM scores to produce targeted feedback; the Refiner revises chains using that feedback. The review-refine cycle iterates until a condition passes or the maximum iteration is reached. Each iteration keeps the top k of 2k chains by ORM score, and final answers are selected by Weighted Self-Consistency using the sum of ORM and PRM solution-level scores.

Limitations

  • The paper states that MAGICORE increases inference compute, relies on an initial base number of samples to establish difficulty and solution quality, and requires ORMs and PRMs that must be separately trained for a given domain and therefore do not exist for all problem types. It notes the framework is modular, so newer and better ORMs and PRMs can be swapped in, and says it has no additional risks beyond those inherent to LLMs generally.
  • Evident from the text: evaluation is limited to English datasets, and commonsense/logical experiments use GPT4o-mini as a PRM because existing standalone PRMs generally only exist for math. The paper also states it does not compare to MCTS-based methods because their multiple rollouts make generation-matched comparisons challenging.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2025.emnlp-main.1660

Cites

0 citation errors · 5 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 17 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Uniformly refining all instances can cause over-refinement, where solutions that were already correct before refinement are flipped to incorrect, reducing overall performance.

    “Uniformly refining all instances can cause over-correction and reduce overall performance.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Huang et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Shridhar et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Stechly et al., 2024 source
  • Premisecitation

    Recent work shows the inability of LLMs to self-verify their own reasoning.

    “However, recent work shows the inability of LLMs to”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Huang et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Stechly et al., 2023 source

      The Introduction and Conclusion report that LLMs were poor at verifying solutions and that self-critiquing did not improve results. The experiments concerned GPT-4 on graph-coloring problems, so they do not establish a general inability of LLMs to self-verify reasoning across tasks. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Kamoi et al., 2024

      Section 7 (p. 10) says LLMs struggle to generate reliable feedback on their own responses without additional information. But the paper limits this difficulty to certain conditions and says verification can be easy for some tasks (Section 5.1, p. 8), so an unqualified inability to self-verify reasoning overstates its conclusion. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingown data

    MAGICORE shows consistent improvements over all aggregation and refinement baselines across datasets and models.

    “Notably, MAGICORE shows consistent improvements over all aggregation and refinement baselines across datasets and models.”
    Quote in the paperSubstantial claim
  • Findingown data

    One iteration of selective refinement results in up to 2.6% improvement compared to uniformly applying aggregation.

    “Conversely, one iteration of our selective refinement (row 3) targets only the challenging instances where the weighted majority vote is unlikely to succeed, resulting in up to 2.6% improvement compared to uniformly applying aggregation (row 1).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The difficulty classification thresholds (S_avg ≥ 0 and entropy ≥ 0.5) correctly distinguish easy from hard problems.
  • Findingown data

    MAGICORE shows increased performance across iterations even as baselines stagnate.

    “We also highlight the role of iteration in our framework, showing increased performance across iterations even as baselines stagnate.”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 4
  • Classified as probably not a substantial claim. 11
  • May combine two claims. 34

  • The quote only partly states this claim.

    Findingown data

    One iteration of MAGICORE outperforms Self-Consistency by 3.4% when Self-Consistency uses k=120.

    “just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2% and Self-Consistency (Wang et al., 2022) by 3.4%”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang et al., 2022 source

  • Classified as probably not a substantial claim.

    Premisecitation

    Refinement, where solutions are critiqued and improved upon during resampling, offers a possible avenue for breaking out of the aggregation rut.

    “Refinement -where solutions are instead critiqued and improved upon during resamplingoffers a possible avenue for breaking out of the aggregation rut.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Madaan et al., 2023 source

  • The quote only partly states this claim.

    Findingcitation

    One iteration of MAGICORE outperforms Best-of-k by 3.2% when Best-of-k uses k=120.

    “just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2%”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Lightman et al., 2023 source

  • The quote only partly states this claim.

    Findingown data

    One iteration of MAGICORE outperforms Self-Refine by 4.0%.

    “MAGICORE also outperforms a combination of Self-Refine (Madaan et al., 2023) and Self-Consistency by 4.0%”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Madaan et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Self-Consistency generates k reasoning chains and marginalizes over the chains to obtain answer clusters.

    “Self-Consistency (SC; Wang et al., 2022) generates k reasoning chains and marginalizes over the chains to obtain answer clusters”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    MAGICORE is complementary to and enhanced by progress in reward modeling, rather than constrained by it.

    “the community is actively advancing the performance of reward models evidenced by a benchmark for reward models (Lambert et al., 2024), MAGICORE is thus complementary to and enhanced by progress in reward modeling, rather than constrained by it.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lambert et al., 2024 source

  • May combine two claims.

    Findingnone given

    Applying aggregation strategies uniformly may waste computation on easier problems where performance saturates quickly, and has diminishing gains on harder problems even when more samples are generated.

    “However, applying aggregation strategies uniformly may waste computation on easier problems where the performance saturates quickly, and has diminishing gains on the harder problems even when more samples are generated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MAGICORE scales better with more iterations of refinement, scales to stronger base models and RMs, applies to both math/reasoning and commonsense tasks, and continues to improve while the baselines stagnate.

    “MAGICORE also scales better with more iterations of refinement, scales to stronger base models and RMs, applies to both math/reasoning and commonsense tasks, and continues to improve while the baselines stagnate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAGICORE shows 17.1% and 13.5% improvements over Self-Refine for Llama-3-8B and GPT-3.5, respectively.

    “On average, MAGICORE shows 17.1% and 13.5% improvements over SR for Llama-3-8B and GPT-3.5.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAGICORE with k = 15 already performs better than Best-of-120 and 120-way SC.

    “Notably, MAGICORE with k = 15 already performs better than Best-of-120 and 120-way SC, highlighting the cost-effectiveness of our method.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAGICORE consistently outperforms both coarse-grained aggregation and fine-grained refinement alone at any given budget, and even outperforms baselines using substantially more computation.

    “Results across five math datasets and two models show that our coarse-to-fine method consistently outperforms both coarse-grained aggregation and fine-grained refinement alone at any given budget, and even outperforms baselines using substantially more computation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Different problems require varying amounts of computation.

  • May combine two claims.

    Findingown data

    Replacing actual PRM scores with random scores performs worse than using PRM scores, indicating that PRM scores help in localizing errors.

    “in row 2, we replace the actual PRM scores with random scores.”
    • Also stated in: § Method ARC Date · hedged
      “worse than row 4, indicating that PRM scores help in localizing errors.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    MAGICORE's first iteration involves 40 samples for easy problems and 40 refined chains for the subset of hard problems, making k = 55 on average.

    “Note that our method's first iteration only involves 40 samples for easy problems and 40 refined chains for the subset of hard problems, making our k = 55 on average.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAGICORE transfers to commonsense and logical reasoning, outperforming 120-way SC by 2.5% and 8.0%, respectively.

    “Table 7 shows that MAGICORE transfers to commonsense and logical reasoning, outperforming 120-say SC by 2.5% and 8.0%, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Existing standalone PRMs generally only exist for math.

  • May combine two claims.

    Findingreasoning

    Baselines like Best-of-k and Self-Consistency stagnate with additional compute, while MAGICORE continues to improve.

    “indeed, while baselines like Best-of-k and Self-Consistency stagnate with additional compute, MAGICORE continues to improve.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAGICORE improves stronger models like GPT4o-mini, though with a smaller margin than for Llama3-8B and GPT-3.5.

    “MAGICORE also improves stronger models like GPT4o-mini.”
    • Also stated in: § MAGICORE generalizes to other domains. · boosted
      “Table 8 demonstrates that MAGICORE can also enhance stronger model's performance, albeit with a smaller margin of improvement compared to Llama3-8B and GPT-3.5 shown in Table 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    SC saturates and stops improving at around k = 40, while MAGICORE continues to improve with increasing k.

    “Moreover, while SC saturates and stops improving at around k = 40, MAGICORE continues to improve with increasing k.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using LLM self-refinement results in an average drop of 1.5% compared to MAGICORE, indicating that using the LLM for refinement is less effective than using a PRM.

    “Compared to MAGICORE in row 4 of Table 6, using LLM's self-refinement (row 1) results in an average drop of 1.5%, indicating that using the LLM for refinement is less effective than using a PRM.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Compared to 120-way SC, MAGICORE shows an average improvement of 3.3% on Llama-3-8B and 3.2% on GPT-3.5.

    “When compared to 120-way SC, our method shows an even greater average improvement of 3.3% on Llama-3-8B and 3.2% on GPT-3.5.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAGICORE stabilizes at the third iteration with 75.6% accuracy, a 1.4% improvement compared to the first iteration, while SR + SC fluctuates around 70%.

    “while SR + SC fluctuates around the same range of 70%, MAGICORE continues to improve and stabilize at the third iteration with 75.6% accuracy (with a noticeable 1.4% improvement compared to the first iteration).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    MAGICORE effectively decides when to use refinement and when not to, leading to a 6.4% improvement over the strongest Best-of-k baseline on MATH.

    “Moreover, MAGICORE effectively decides when to use refinement and when not to, leading to a 6.4% improvement over the strongest Best-of-k baseline on MATH (Hendrycks et al., 2021b)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hendrycks et al., 2021b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The difficulty classification thresholds (S_avg ≥ 0 and entropy ≥ 0.5) correctly distinguish easy from hard problems.

  • May combine two claims.

    Findingown data

    A single iteration of MAGICORE already outperforms methods that rely on PRM for aggregation (120-way SC + PRM), as well as Self-Correction, advanced prompting, and multi-agent debate.

    “Results show that a single iteration of MAGICORE already outperforms methods that rely on PRM for aggregation (120-way SC + PRM), as well as approaches like Self-Correction, advanced prompting, and multi-agent debate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On MMLU, MATH and SAT, MAGICORE consistently improves with additional tokens, unlike Self-Consistency which tends to stagnate.

    “On MMLU, MATH and SAT, we observe a clear upward trend with an increased token count; MAGI-CORE consistently improves with additional tokens (unlike SC which tends to stagnate).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Scaling Self-Consistency from k = 40 to k = 120 introduces substantial token overhead while providing marginal improvements.

    “Scaling Self-Consistency from k = 40 to k = 120 introduces substantial token overhead while providing marginal improvements.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Prompt-based methods perform worse than the random baseline, particularly the confidence-score method which classifies all problems as easy and scores 0 precision and recall.

    “Interestingly, the prompt-based methods perform worse than the random baseline, particularly the one relying on confidence scores, which classifies all problems as easy; this method scores 0 for both precision and recall since we treat”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The first iteration of MAGICORE consistently outperforms 120-way SC while using fewer tokens.

    “Notably, the first iteration of MAGICORE consistently outperforms 120-way SC while using fewer tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Scaling Self-Consistency from k = 40 to k = 120 largely increases token overhead while yielding marginal improvements.

    “Results in Fig. 5 show that (1) scaling Self-Consistency from k = 40 to k = 120 largely increases token overhead while yielding marginal improvements.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    MAGICORE's reward models can be swapped in without modification.

    “Performance of MAGICORE with different RMs, which can be swapped in without modification.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    While each condition individually outperforms LLM self-verification, combining both conditions yields the best performance.

    “Results indicate that while each condition individually outperforms LLM self-verification, combining both yields the best performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Oracle PRM score performs best, followed by predicted PRM score.

    “Appendix D shows that the oracle PRM score performs the best, followed by the predicted PRM score”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Maintaining separate Reviewer and Refiner roles leads to better performance than combining them, with Joint Roles resulting in a 0.6% drop in MMLU and a 1.2% decrease in MATH.

    “Our findings show that maintaining separate roles (as in our multi-agent setup) leads to better performance, with the”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    MAGICORE relies on starting with a base number of samples to establish the difficulty and quality of existing solutions.

    “and relies on starting with a base number of samples to establish the difficulty and quality of existing solutions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Adaptively addressing challenging instances with targeted refinement improves overall performance while reducing compute for easy problems.

    “This suggests that adaptively addressing challenging instances with targeted refinement improves overall performance, while reducing compute for easy problems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    For some hard instances, one round of refinement may be insufficient, as the Reviewer may have generated some irrelevant feedback or the Refiner may not have fixed the highlighted step adequately.

    “For some hard instances, one round of refinement may be insufficient, as the Reviewer may have generated some irrelevant feedback or the Refiner may not have fixed the highlighted step adequately (”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    MAGICORE is designed to improve the reasoning of LLMs, and thus has no additional risks beyond those inherent to LLMs generally.

    “MAGICORE is designed to improve the reasoning of LLMs, and thus has no additional risks beyond those inherent to LLMs generally.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    MAGICORE is modular and allows newer and better ORMs and PRMs to be swapped in as they become available.

    “However, we also note that MAGICORE is modular, and thus allows for newer and better ORMs and PRMs to be swapped in as they become available.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs struggle to identify their own mistakes (steps needing refinement) and struggle to correct them without explicit instructions.

    “LLMs struggle to identify and correct their own mistakes.”
    • Also stated in: § Refined Solution · plain
      “LLMs struggle to identify their own mistakes (i.e., steps needing refinement) and struggle to correct them without explicit instructions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tyen et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Selecting the solution via another scoring model (ORM) while using PRM for feedback avoids overfitting.

    “Here we choose to base the decision on the ORM score because the targeted feedback is generated with PRM's step-wise scores, so selecting the solution via another scoring model avoids overfitting.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    LLMs can resolve errors when explicitly pointed out.

    “Inspired by the finding that LLM can resolve errors when explicitly pointed out (Tyen et al., 2024)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tyen et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisenone given

    Stopping refinement early could leave errors unaddressed; hard problems might be underthought by a single refinement iteration.

    “Stopping refinement too soon could leave errors unaddressed.”
    • Also stated in: § Refined Solution · hedged
      “stopping refinement early could leave errors unaddressed, i.e., hard problems might be”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    The performance of adaptive sampling approaches is upper-bounded by Self-Consistency; they address cost but do not enhance overall performance.

    “However, the performance of such approaches is upper-bounded by SC -they address the cost issue but do not enhance overall performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Selective refinement avoids over-correction and enhances overall performance by effectively allocating more resources to harder problems.

    “This demonstrates that our selective refinement not only avoids over-correction but also enhances overall performance by effectively allocating more resources to harder problems.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The difficulty classification thresholds (S_avg ≥ 0 and entropy ≥ 0.5) correctly distinguish easy from hard problems.

  • May combine two claims.

    Interpretationreasoning

    MAGICORE's multi-agent refinement incorporates external RMs for more objective scoring, enabling the generation of targeted feedback for better refinement.

    “Therefore, MAGICORE's multi-agent refinement incorporates external RMs for more objective scoring, enabling the generation of targeted feedback for better refinement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: A major challenge in multi-agent systems is achieving a correct consensus among LLMs.
    • Rests on an assumption: External feedback can help prevent consensus from aligning with agents' internal and possibly erroneous outputs.

  • May combine two claims.

    Interpretationnone given

    MAGICORE adapts resource allocation based on problem difficulty and refines only when encountering hard problems to avoid excessive refinement.

    “MAGICORE adapts resource allocation based on problem difficulty and refines only when encountering hard problems to avoid excessive refinement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Three main issues in refinement are excessive refinement, inability to localize and address errors, and insufficient refinement.
44 of 44

  • The paper relies on this without support.

    The difficulty classification thresholds (S_avg ≥ 0 and entropy ≥ 0.5) correctly distinguish easy from hard problems.

    “If S avg ≥ 0 after normalization, this condition will be true, meaning the majority answer is already high-quality”
    “≥ 0.5 < 0.5 High Low”

  • The paper relies on this without support.

    PRM step-wise scores identify steps that are likely to be incorrect.

    “we leverage step-by-step scores from a process reward model (PRM) to help the LLM pinpoint low-scoring steps (which are likely to be incorrect)”

  • The paper relies on this without support.

    The Reviewer can generate actionable feedback that localizes errors when supplied with step-wise PRM scores.

    “The goal of the Reviewer is to incorporate these step-wise correctness scores to generate actionable feedback.”

  • The paper relies on this without support.

    Different problems require varying amounts of computation.

    “Building on the observation that different problems require varying amounts of computation”

  • The paper relies on this without support.

    Existing standalone PRMs generally only exist for math.

    “as existing standalone PRMs generally only exist for math.”

  • The paper relies on this without support.

    A major challenge in multi-agent systems is achieving a correct consensus among LLMs.

    “a major challenge in multi-agent systems is achieving a correct consensus among LLMs”

  • The paper relies on this without support.

    External feedback can help prevent consensus from aligning with agents' internal and possibly erroneous outputs.

    “external feedback can help prevent this consensus from aligning with the agents' internal and possibly erroneous outputs.”

  • The paper relies on this without support.

    Reward models must be separately trained to provide rewards for a given domain and therefore do not exist for all problem types.

    “These models must be separately trained to provide rewards for a given domain and therefore do not exist for all problem types.”

  • The paper relies on this without support.

    Three main issues in refinement are excessive refinement, inability to localize and address errors, and insufficient refinement.

    “Top: Three main issues in refinement: (1) Excessive refinement; (2) Inability to localize and address errors; (3) Insufficient refinement.”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • MMLU gain over Best-of-k is understated

    Major

    For Llama-3-8B, the third-iteration MMLU score is 68.9 and the Best-of-k score is 62.6, a gain of 6.3 points rather than the reported 5.6.

    Reported 5.6 · recomputed 6.3

  • SVAMP gain over Best-of-k is understated

    Major

    For Llama-3-8B, the third-iteration SVAMP score of 91.3 exceeds the Best-of-k score of 88.7 by 2.6 points, not 1.7.

    Reported 1.7 · recomputed 2.6

  • MATH gain over Best-of-k is understated

    Major

    For Llama-3-8B, 47.8 minus the Best-of-k score of 41.4 is 6.4 points, not the reported 5.2.

    Reported 5.2 · recomputed 6.4

  • Average gain over Best-of-k does not match table scores

    Minor

    The displayed Llama-3-8B averages are 75.6 for the third iteration and 71.0 for Best-of-k. Their difference is 4.6 points, rather than the reported 4.3.

    Reported 4.3 · recomputed 4.6

4 of 4

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021a. Measuring massive multitask language understanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
  2. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315-5333.
  3. Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan- Hung Yu, Hung yi Lee, and Shao-Hua Sun. 2024. LLM discussion: Enhancing the creativity of large language models via discussion framework and role- play. In First Conference on Language Modeling.

Cited by

Sorted by citations

Authors
Chen, J., Prasad, A., Saha, S., Stengel-Eskin, E., & Bansal, M.
Affiliations
University of North Carolina at Chapel Hill (US)
ORCID
Venue
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2025
Pages
32651-32674
DOI
10.18653/v1/2025.emnlp-main.1660
OpenAlex
W4416035350
Cited by
3
References
0
Citations by year
2026: 2 · 2025: 1
FWCI
3.24
Type
conference paper
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language model reasoning, coarse-to-fine refinement, self-refinement, overcorrection, error localization, self-consistency, computational efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review