Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

Existing natural-language-feedback methods mainly use supervised fine-tuning to imitate static, pre-collected critiques or critique-guided refinements.

“However, existing methods primarily utilize supervised fine-tuning (SFT) to imitate static, pre-collected critiques or critique-guided refinements (Wang et al., 2025;Chen et al., 2024;Xi et al., 2024).”

• cites Wang et al., 2025 sourcePartly supported

  The Introduction says SFT trains models to imitate annotated responses, supporting the claim's point about imitation. But it does not say that existing natural-language-feedback methods primarily imitate static, pre-collected critiques or critique-guided refinements. The paper instead presents training on critiques as its proposed alternative to traditional SFT (Introduction; page unavailable). (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Chen et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Xi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The effectiveness of critiques extends from simple indicators to detailed chain-of-thought evaluations.

“This efficacy extends from simple indicators to detailed chain-of-thought (CoT) evaluations (Wang et al., 2025).”

• cites Wang et al., 2025 sourcePartly supported

  Section 3 (Experiments; page unavailable) reports improved benchmark results for critique fine-tuning, and Appendix A.4.1 shows detailed, step-by-step critique prompts. Our copy does not show that the paper tested the effectiveness of simple indicators or established efficacy across the claimed range of critique formats. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Reinforcement learning with numerical feedback has been highly effective for improving LLM reasoning.

“Reinforcement learning with numerical feedback has proven highly effective for improving the reasoning capabilities of LLMs (Ope-nAI et al., 2024;DeepSeek-AI et al., 2025;Fatemi et al., 2025;Li et al., 2025;Liu et al., 2025a;Yu et al., 2025).”

• cites OpenAI et al., 2024Could not verify

  Section 2 (Model data and training, pp. 1–2) says code and math training data help develop reasoning skills, and mentions post-training alignment to human preferences. Searches for reinforcement learning, numerical feedback, and rewards found no passage connecting that method to improved LLM reasoning. Our copy is truncated, so the claim cannot be verified from it. (LLM, conf=0.70)

  Experimental — accuracy not yet measured
• cites DeepSeek-AI et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fatemi et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Reinforcement learning with numerical rewards has substantially improved the complex reasoning abilities of large language models.

“Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs).”

• Also stated in: § 1. Introduction · plain

  “RL has been a key driver of recent advancements in improving the reasoning capabilities of LLMs (Yang et al., 2025a;DeepSeek-AI et al., 2025;OpenAI et al., 2024;Ope-nAI, 2025).”

• cites Yang et al., 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites DeepSeek-AI et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites OpenAI et al., 2024Could not verify

  Section 2 (Model data and training) describes aligning the model to human preferences during post-training, but does not say that reinforcement learning with numerical rewards improved complex reasoning. Searches for “reinforcement learning,” “numerical rewards,” and reasoning improvements found no passage addressing that claim. Our copy is truncated, so relevant text may be missing. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Recent work that combines online RL with expert demonstrations is limited by its need for curated, high-quality data.

“To bridge this gap, recent work has combined online RL with expert demonstrations (Yan et al., 2025;Lv et al., 2025;Lanchantin et al., 2025); however, these approaches are constrained by their reliance on curated, high-quality data.”

• cites Yan et al., 2025 sourcePartly supported

  Section 2 describes combining on-policy RL with reasoning traces from a stronger model, and Section 3 (Dataset Construction) reports filtering 94k prompts and traces down to 45k. The paper also says it leverages high-quality off-policy data, but it does not characterize that reliance as a constraint. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Lv et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lanchantin et al., 2025 sourceCould not verify

  The Conclusion (p. 10) compares offline, semi-online, and online RL methods, but does not describe combining online RL with expert demonstrations. The Introduction (p. 1) and the additional passages searched for “expert,” “demonstrations,” and “curated” do not establish a reliance on curated, high-quality data or identify it as a limitation. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Verbal credit assignment from critiques lets models reach high-quality refinement trajectories through in-context learning, which standard scalar-reward exploration cannot reach.

“This suggests that verbal credit assignment from critiques enables models to access high-quality refinement trajectories via in-context learning (Liu et al., 2022)-trajectories unreachable via standard exploration with scalar rewards alone.”

• cites Liu et al., 2022 sourceCould not verify

  The Trial and Fine-tuning sections discuss selecting retrieved examples to improve GPT-3’s in-context learning. Our copy does not address verbal credit assignment from critiques, refinement trajectories, or a comparison with scalar-reward exploration; searches for those terms found no supporting passage. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

GRPO removes the need for a value function approximation by estimating advantages from the relative performance of a group of sampled responses.

“Unlike Proximal Policy Optimization (PPO) (Schulman et al., 2017), it eliminates the need for a value function approximation by estimating advantages based on the relative performance of a group of sampled responses for the same query.”

• cites Schulman et al., 2017 sourcePartly supported

  Section 5 (pp. 4–5) describes PPO implementations that use a learned state-value function and a value-function loss. It says most variance-reduced advantage estimators use a value function, not that PPO requires one; the claim’s implication of a strict requirement is therefore unsupported. The GRPO mechanism is not attributed to this PPO paper. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Clipping is insufficient for off-policy refinements because of vector orthogonality.

“While effective for on-policy data, we prove that clipping is insufficient for off-policy refinements due to vector orthogonality.”

• cites Schulman et al., 2017 sourceSupported

Natural language feedback supplies textual critiques with fine-grained diagnostic signals that support precise refinement of responses.

“Natural Language Feedback (NLF) provides textual critiques with granular diagnostic signals that facilitate precise response refinement (Saunders et al., 2022;Chen et al., 2024;Zhang et al., 2026).”

• cites Saunders et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chen et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhang et al., 2026 sourcePartly supported

  Section 3.3 says RETROAGENT distills past experiences into actionable textual lessons that guide the agent on how to improve. The appendix’s lesson-quality criteria call for specific errors and their root causes, supporting the diagnostic aspect. The paper applies this feedback to an agent’s subsequent decisions, however, rather than establishing precise refinement of responses generally. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

CoT critiques successfully refine 55.37% of persistently failed questions for Qwen2.5-7B-Base.

“successfully refine 55.37% of persistently failed questions for Qwen2.5-7B-Base”

Reinforcement learning that relies only on numerical feedback leads to performance plateaus.

“performance plateaus”

• Also stated in: § 1. Introduction · plain

  “(i) Performance Plateaus: Performance stagnates despite 8× training data scaling or extended training.”

Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods across eight reasoning tasks.

“Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instructacross eight challenging reasoning tasks.”

Critique-GRPO enables effective self-improvement through self-critiquing.

“Notably, Critique-GRPO facilitates effective selfimprovement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024.”

All three critique types enable successful refinement of previously unsolvable problems.

“All three critique types enable successful refinement of previously unsolvable problems”

Even minimal language feedback can catalyze error correction where scalar rewards failed.

“Our key findings, summarized below, demonstrate that even minimal language feedback can catalyze error correction where scalar rewards failed.”

The best RL-finetuned Qwen2.5-7B-Base consistently fails on about 29% of training questions with Pass@4 equal to zero.

“As shown in Table 1 (left panel), even the best-performing RLfinetuned Qwen2.5-7B-Baseconsistently fails on approximately 29% of training questions (Pass@4 = 0)”

Reinforcement learning that relies only on numerical feedback leads to persistent failures.

“(iii) Persistent Failures: Models consistently fail on specific training problems despite extensive iterations.”

CoT critiques achieve the highest valid refinement rate, 36.47%.

“Table 1 shows that CoT critiques achieve the highest valid refinement rate (36.47%)”

Critique-GRPO achieves average Pass@1 improvements of about +15.0 to +21.6% on various Qwen models across eight reasoning tasks.

“achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models”

Self-reflection behaviors contribute minimally to problem-solving across both reasoning tasks.

“Figure 6a (Appendix E) shows self-reflection behaviors contribute minimally across both reasoning tasks.”

RL models whose performance has plateaued can successfully correct failed solutions when given natural language critiques.

“We show that plateaued RL models can successfully refine failed solutions when given natural language critiques.”

• Also stated in: § 3.2. Promise of Learning from Natural Language Feedback · boosted

  “All three critique types enable successful refinement of previously unsolvable problems, demonstrating that deliberate critiques succeed where spontaneous selfreflection fails.”

Critique-GRPO achieves an average Pass@1 improvement of +7.3% on Llama-3.2-3B-Instruct across eight reasoning tasks.

“and +7.3% on Llama-3.2-3B-Instructacross eight challenging reasoning tasks”

Reinforcement learning that relies only on numerical feedback leads to ineffective spontaneous self-reflection.

“(ii) Ineffective Spontaneous Self-Reflection: Spontaneous “Aha moments” (DeepSeek-AI et al., 2025) rarely improve problem-solving success.”

Deliberate critiques enable successful refinement where spontaneous self-reflection fails.

“Deliberate critiques outperform spontaneous selfreflection.”

• Also stated in: § 3.2. Promise of Learning from Natural Language Feedback · boosted

  “All three critique types enable successful refinement of previously unsolvable problems, demonstrating that deliberate critiques succeed where spontaneous selfreflection fails.”

Qwen2.5-7B-Base performance saturates after 120 steps.

“As shown in Figure 5a (Appendix E), Qwen2.5-7B-Baseperformance saturates after 120 steps”

Self-critiquing Critique-GRPO achieves a +16.7% Pass@1 improvement over GRPO on AIME 2024.

“achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024”

Critique-GRPO is robust to diverse critique sources.

“Notably, the framework is robust to diverse critique sources and ex-hibits strong self-improvement capabilities”

Removing KL regularization yields a +1.5% gain.

“removing KL (+1.5%)”

For Qwen2.5-7B-Base, 87 unsolved problems from Minerva-Math and 37 from GPQA-Diamond were identified.

“For Qwen2.5-7B-Base, we identify 87 unsolved problems from the Minerva-Math dataset and 37 from the GPQA-Diamond dataset.”

Critique-GRPO with weaker refinement via critique with ground truth outperforms R1-GRPO, 65.55% versus 63.75%.

“and outperforms R1-GRPO (65.55% v.s.63.75%)”

• AssumptionThe authors state this as an assumption: Qwen3-8B-Base is a weaker teacher than Qwen3-8B.

Critique-GRPO with self-critique improves average pass@1 by 4.5% over R1-GRPO.

“Table 4 shows that Critique-GRPO (selfcritique) improves average pass@1 by 4.5% over R1-GRPO”

• Also stated in: § H. Detailed Results and Analysis of Self-Critiquing Mechanisms · plain

  “On average, Critique-GRPO (self-critique) improves pass@1 by +4.5% and +12.0%compared to R1-GRPO and SFT, respectively.”

Critique-GRPO with weaker refinement via critique with ground truth achieves a +12.3% average pass@1 improvement over Qwen3-8B.

“As shown in Table 7, Critique-GRPO (weaker refinement via critique with ground truth) achieves a +12.3% average pass@1 improvement over Qwen3-8B”

• AssumptionThe authors state this as an assumption: Qwen3-8B-Base is a weaker teacher than Qwen3-8B.

Critique-GRPO achieves a 21.6% average pass@1 improvement over Qwen2.5-Math-7B-Base using only 4k RL training prompts.

“Table 3 shows that Critique-GRPO achieves a 21.6% average pass@1 improvement over Qwen2.5-Math-7B-Base using only 4k RL training prompts”

Critique-GRPO with self-critique improves average pass@1 by 12.0% over SFT.

“and 12.0% over SFT”

• Also stated in: § H. Detailed Results and Analysis of Self-Critiquing Mechanisms · plain

  “On average, Critique-GRPO (self-critique) improves pass@1 by +4.5% and +12.0%compared to R1-GRPO and SFT, respectively.”

For the non-reasoning model in the mathematical domain, self-reflection behaviors barely contribute.

“In Figure 6a, self-reflection behaviors barely contribute in the mathematical domain for the non-reasoning model.”

Critique-GRPO with CoT critique surpasses Refinement FT by +11.9 points in average pass@1 on Qwen2.5-7B-Base.

“Critique-GRPO (CoT critique) surpasses Refinement FT by +11.9 points (47.08% vs. 35.21%)”

Critique-GRPO with CoT critiques consistently outperforms its indicative critique variant, with average pass@1 improvements of +1.8 to +2.4% on Qwen2.5-7B-Base.

“Critique-GRPO with CoT critiques consistently outperforms its indicative critique variant, achieving average pass@1 improvements of +1.8-2.4% on Qwen2.5-7B-Base”

Increasing critique token length to around 2079 tokens causes a slight performance drop from 47.08 to 46.00.

“However, increasing the token length to around 2079 tokens by prompting the model to produce highly detailed critiques results in a slight performance drop (from 47.08 to 46.00).”

Policy shaping yields a +3.1% gain to 47.08%.

“policy shaping (+3.1% to 47.08%)”

Persistent failures occurred even though the models underwent trial-and-error fine-tuning that rewarded correct responses and penalized incorrect ones.

“This occurred despite undergoing trial-and-error fine-tuning, where correct responses are rewarded, and incorrect responses are penalized.”

Shorter CoT critiques of about 683 tokens generally achieve strong performance.

“As shown in Table 10, shorter CoT critiques (approximately 683 tokens) generally achieve strong performance.”

Critique-GRPO advances AMC23 accuracy to 62.50%.

“markedly surges to 62.50% on AMC23”

Qwen2.5-7B-Base reaches its highest performance at about 45-46% accuracy after 120 steps.

“Qwen2.5-7B-Basereaches its highest performance at approximately 45-46% accuracy after 120 steps (Figure 5a)”

All results using Critique-GRPO consistently surpass those of GRPO.

“Despite this, all results using Critique-GRPO consistently surpass those of GRPO, demonstrating the robustness of Critique-GRPO.”

Critique-GRPO with self-critique consistently outperforms R1-GRPO and SFT across pass@k metrics on AIME24 and AIME25 for k from 1 to 256.

“Figures 3 and 9b demonstrate that Critique-GRPO (self-critique) consistently outperforms R1-GRPO and SFT across pass@k metrics on AIME24 and AIME25 for k ∈ [1, 256]”

Adding language feedback yields a +0.6% gain to 43.26%.

“adding language feedback (+0.6% to 43.26%)”

For the reasoning model, self-reflection behaviors make a noticeable contribution.

“the reasoning model shows that self-reflection behaviors make a noticeable contribution (Figure 6b).”

CoT critiques facilitate effective model refinement.

“CoT Critiques facilitate effective model refinement.”

Self-correction-related behaviors such as backtracking, backward chaining, and anticipation still contribute considerably less.

“Nevertheless, self-correction-related behaviors, such as backtracking, backward chaining, and anticipation, still contribute considerably less.”

Critique-GRPO with CoT critique surpasses Refinement FT by +8.81 points in average pass@1 on Qwen3-8B.

“and +8.81 points (68.26% vs. 59.45%) in average pass@1 on Qwen2.5-7B-Base and Qwen3-8B, respectively”

Peak accuracy shows negligible improvement despite an eightfold increase in training prompts or extending training to 420 steps.

“Critically, peak accuracy shows negligible improvement despite an 8-fold increase in training prompts or extending the training duration to 420 steps.”

Critique-GRPO advances MATH-500 accuracy to 77.80%.

“Critique-GRPO advances the MATH-500 accuracy to 77.80%”

Early entropy peaks before step 200 occur when self-generated refinements deviate significantly from initial responses, increasing entropy and enabling beneficial distributional shifts.

“Early entropy peaks (before step 200) occur when self-generated refinements deviate significantly from initial responses, increasing entropy and enabling beneficial distributional shifts.”

Quality-based selection yields a +0.7% gain to 43.95%.

“quality-based selection (+0.7% to 43.95%)”

Critique-GRPO with CoT critique exceeds CITL-FT by +12.4 points on Qwen3-8B.

“and +12.4 points (68.26% vs. 55.84%) on Qwen3-8B”

Critique-GRPO consistently outperforms both supervised and RL-based fine-tuning methods on Qwen2.5-7B-Base and Qwen3-8B across all tasks.

“Critique-GRPO consistently outperforms both supervised and RL-based fine-tuning methods on Qwen2.5-7B-Base and Qwen3-8B across all tasks.”

Qwen3-8B plateaus at 65-67% accuracy after 200 steps.

“while Qwen3-8B plateaus at 65-67% accuracy after 200 steps (Figure 5b)”

RL with solely numerical feedback frequently encounters performance plateaus.

“RL with solely numerical feedback frequently encounters performance plateaus.”

Deliberate critique is more effective than spontaneous self-reflection for self-correction.

“Deliberate critique is more effective than spontaneous self-reflection for self-correction.”

Critique-GRPO improves average pass@1 by +4.4 points over R1-GRPO and R1-Dr.GRPO on Qwen2.5-7B-Base.

“Compared to R1-GRPO and R1-Dr.GRPO, Critique-GRPO improves average pass@1 by +4.4 points (42.66% → 47.08%) on Qwen2.5-7B-Base”

Critique-GRPO with CoT critique exceeds CITL-FT by +11.4 points on Qwen2.5-7B-Base.

“It also exceeds CITL-FT by +11.4 points (47.08% vs. 35.66%)on Qwen2.5-7B-Base”

Critique-GRPO substantially outperforms numerical-feedback approaches that require 46k prompts.

“substantially outperforming numericalfeedback approaches that require 46k prompts”

Critique-GRPO improves average pass@1 by +3.8 points over R1-GRPO and R1-Dr.GRPO on Qwen3-8B.

“and +3.8 points (64.46% → 68.26%) on Qwen3-8B”

Experiments across eight reasoning tasks show that Critique-GRPO consistently outperforms existing SFT and numerical-feedback-only RL methods.

“Experiments across eight reasoning tasks demonstrate that Critique-GRPO consistently outperforms existing SFT and numerical-feedback-only RL methods”

Inverting feedback conclusions causes entropy explosion and subsequent performance degradation.

“For example, by inverting feedback conclusions (e.g., swapping “correct” and “incorrect”), we observe entropy explosion and subsequent performance degradation.”

Critique-GRPO consistently surpasses GRPO across all eight reasoning tasks on Qwen3-32B.

“As shown in Table 11, Critique-GRPO consistently surpasses GRPO across all eight reasoning tasks.”

Critique-GRPO outperforms R1-GRPO in terms of factuality and correctness.

“Additionally, the superior performance of RL fine-tuning with Critique-GRPO over R1-GRPO in terms of factuality and correctness indicates that CoT critiques help the model effectively identify errors and improve valid exploration.”

For Qwen3-8B-Base, 50 previously unsolved problems from Minerva-Math and 15 from GPQA-Diamond were identified.

“We identify 50 previously unsolved problems from the Minerva-Math dataset and 15 from the GPQA-Diamond dataset for Qwen3-8B-Base.”

Critique-GRPO with self-critique and self-evaluation improves average pass@1 by 2.3% over R1-GRPO.

“Additionally, the unsupervised approach-Critique-GRPO (self-critique & self-evaluation)-achieves an average pass@1 improvement of 2.3% over R1-GRPO, highlighting the potential of leveraging self-critique for self-improvement without any external supervision.”

The base Qwen3-8B model gave a wrong answer to the math problem because it wrote the expression as sin(A+B) • sin(A-B).

“The base Qwen3-8B model produces an incorrect response due to an incorrect formulation of the expression as sin(A + B) • sin(A -B), as shown in the red square.”

CoT critique refinement significantly outperforms refinement based on indicative critiques or critiques with ground truth.

“This performance significantly surpasses refinement based on indicative critiques or critiques with ground truth, even though the CoT critique generation process produces valid critiques for only 60.06% and 50.17% of erroneous responses.”

The CoT critique generation process produces valid critiques for only 60.06% and 50.17% of erroneous responses.

“even though the CoT critique generation process produces valid critiques for only 60.06% and 50.17% of erroneous responses.”

On Qwen3-8B, Critique-GRPO tends to shorten responses.

“Additionally, Critique-GRPO tends to reduce response length on Qwen3-8B (Figure 11b).”

The best-performing RL-finetuned Qwen3-8B-Base persistently failed on 17.18% of training problems.

“Table 12 reveals that the best-performing RL-finetuned Qwen3-8B-Base persistently failed on 17.18% of training problems.”

Critique-GRPO with self-critique consistently outperforms across pass@k metrics for k from 1 to 256.

“Figure 9 highlights the consistently superior performance of Critique-GRPO (self-critique) across pass@k metrics, with k ranging from 1 to 256, indicating genuine improvements.”

Refinement with indicative critiques with or without ground truth yields some successful refinements at approximately 2% to 4% valid refinements.

“Refinement with indicative critiques with/without ground truth also yields some successful refinements, albeit at a substantially lower rate (approximately 2%-4% valid refinements).”

The RL-finetuned Qwen3-8B model using Critique-GRPO gave a correct answer to the math problem by writing the expression as sin(α+β) • sin(β-α).

“In contrast, the RLfinetuned Qwen3-8B model using Critique-GRPO generates a correct response by correctly formulating the expression as sin(α + β) • sin(β -α), as shown in the green square.”

• Also stated in: § J. Qualitative Analysis · plain

  “It remains on the correct path to solve the problem, exhibits valid self-reflection to validate the answer (e.g.,”

All three types of critiques facilitate the LLM's self-refinements.

“In addition, all three types of critiques facilitate the LLM's self-refinements.”

Binary correctness signals alone can provide refinement benefits.

“Binary correctness signals alone can provide refinement benefits.”

R1-GRPO exhibits entropy collapse, with policy entropy dropping sharply at the start of training and declining monotonically to near zero.

“In contrast, R1-GRPO exhibits entropy collapse, where policy entropy drops sharply at the start of training and continues to decline monotonically to near zero.”

Critique-GRPO's policy entropy generally remains higher than that of R1-GRPO and R1-Dr.GRPO.

“As shown in Figure 10a, the policy entropy of Critique-GRPO generally remains higher than that of R1-GRPO and R1-Dr.GRPO, suggesting more consistent exploration.”

Critique-GRPO with self-critique achieves improvements of 10-16.7% over R1-GRPO for pass@k with k=1 to 4 on AIME24.

“Notably, Critique-GRPO (self-critique) achieves remarkable gains over R1-GRPO for pass@k with k = 1 to 4, yielding improvements of 10-16.7% on AIME24 (Figure 9a).”

For Qwen3-8B-Base on previously failed questions, self-reflection behaviors contribute minimally to successful problem-solving.

“Notably, self-reflection behaviors contribute minimally to successful problem-solving.”

Critique-GRPO raises the average Pass@1 score on Qwen3-8B by about 4.5% across eight reasoning tasks relative to GRPO.

“Critique-GRPO improves the average Pass@1 score on Qwen3-8B by approximately 4.5% across eight reasoning tasks compared to GRPO.”

Critique-GRPO achieved average gains of +6 and +4.5 percentage points across eight tasks for Qwen2.5-7B-Base and Qwen3-8B, respectively, using only 4K prompts.

“This overhead is marginal compared to the significant performance improvement, with average gains of +6 and +4.5 percentage points across 8 tasks, achieved using only 4K prompts.”

Critique-GRPO achieves 66.7% pass@1 on AIME 2024 through self-critiquing.

“Critique-GRPO facilitates self-improvement via self-critiquing, obtaining 66.7% pass@1 on AIME 2024 and consistent gains across pass@k scores (k=1-256) over GRPO.”

Policy models sometimes fail to follow CoT critiques when refining their responses.

“Policy models sometimes fail to follow CoT critiques to refine their responses.”

Their experiments showed that adding expert demonstrations to critiques did not significantly improve performance.

“However, our experiments reveal otherwise.”

• Also stated in: § Additional information · plain

  “This behavior limits the effectiveness of expert demonstrations as critiques.”

The RL-finetuned Qwen3-8B model using Critique-GRPO stayed on the correct solution path.

“It remains on the correct path to solve the problem, exhibits valid self-reflection to validate the answer (e.g.,”

When generating refinements, both pre-trained and alignment-tuned models tend to output conclusive sentences and correct answers instead of detailed step-by-step reasoning.

“Upon analyzing the generated refinements, we observe that both pre-trained models (e.g., Qwen2.5-7B-Base) and alignment-tuned models (e.g., Qwen3-8B) tend to produce conclusive sentences and correct answers as refinements, rather than detailed step-by-step reasoning to derive the correct answer.”

Among the three critique types used, CoT critiques produced the largest refinement benefits.

“with CoT critiques demonstrating the greatest benefits for refinement.”

The base Qwen3-8B model made many redundant and ineffective self-reflection attempts that did not help it find the actual errors or answer correctly.

“Furthermore, the base Qwen3-8B model exhibits numerous redundant and ineffective self-reflection attempts (e.g.,”

The RL-finetuned Qwen3-8B model using Critique-GRPO performed valid self-reflection to check the answer.

“It remains on the correct path to solve the problem, exhibits valid self-reflection to validate the answer (e.g.,”

Among questions evaluated with RL-finetuned Qwen2.5-7B-Base, 29.07% had no correct response in four attempts (Pass@4=0).

“| RL-finetuned Qwen2.5-7B-Base | 29.07 | Indicative Critique Indicative Critique w/ GT CoT Critique | 100.00 100.00 60.06 | 2.09 1.98 36.47 | 100.00 100.00 95.10 | 7.05 6.88 55.37 |”

Critique-GRPO yields consistent gains across pass@k scores from k=1 to 256 compared with GRPO.

“Critique-GRPO facilitates self-improvement via self-critiquing, obtaining 66.7% pass@1 on AIME 2024 and consistent gains across pass@k scores (k=1-256) over GRPO.”

The RL-finetuned Qwen3-8B model using Critique-GRPO showed concise and effective reasoning.

“In contrast, the RLfinetuned Qwen3-8B model using Critique-GRPO demonstrates concise and effective reasoning.”

The base Qwen3-8B model's response was excessively long, exceeding 6000 tokens.

“This results in an excessively long response (over 6000 tokens).”

The persistent failures indicate fundamental limitations of numerical-only feedback.

“indicating fundamental limitations of numerical-only feedback”

Sparse scalar rewards give little diagnostic information about where or why errors occur, making it hard for models to identify and correct specific reasoning failures.

“However, numerical feedback is inherently limited in expressivity; sparse scalar rewards provide little diagnostic information about where or why errors occur, making it difficult for models to identify and correct specific reasoning failures (Xi et al., 2024;Gandhi et al., 2025).”

• cites Xi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gandhi et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The sample efficiency of the critique mechanism can be justified through the Transfer Eluder Dimension framework.

“We justify our critique mechanism's sample efficiency via the Transfer Eluder Dimension framework (Xu et al., 2025).”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Integrating expressive critiques into online RL loops to augment numerical feedback had not been explored.

“Crucially, the integration of expressive critiques into online RL loops-to augment purely numerical feedback-remains unexplored.”

Numerical feedback lacks the expressiveness to explain why a response failed or how to correct it.

“Crucially, numerical feedback inherently lacks the expressivity to explain why a response fails or how to correct it.”

• Also stated in: § 2. Related Work · plain

  “However, numerical feedback is inherently limited in expressivity; sparse scalar rewards provide little diagnostic information about where or why errors occur, making it difficult for models to identify and correct specific reasoning failures (Xi et al., 2024;Gandhi et al., 2025).”

• cites Xi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gandhi et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The effectiveness of CoT critiques can be attributed to their richness, such as providing a step-by-step explanation.

“The effectiveness of CoT critiques can be attributed to their richness, e.g., providing a step-by-step explanation.”

Dense intermediate rewards can partially reduce plateaus and persistent failures through fine-grained credit assignment.

“While dense intermediate rewards (Cui et al., 2025a;Feng et al., 2025) can partially mitigate plateaus and persistent failures through fine-grained credit assignment (Sutton, 1984)”

• cites Cui et al., 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Feng et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sutton, 1984Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Granular diagnostic signals in informative language feedback can exponentially speed learning by reducing search-space complexity.

“This aligns with theoretical work showing that granular diagnostic signals in informative language feedback can exponentially accelerate learning by reducing search-space complexity (Xu et al., 2025).”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Dense intermediate rewards are still not enough to fix ineffective self-reflection.

“they remain insufficient for rectifying ineffective self-reflection”

Natural language feedback is a promising way to provide explicit guidance.

“Natural Language Feedback (NLF) (e.g., textual critiques) offers a promising solution by providing explicit guidance (Saunders et al., 2022).”

• cites Saunders et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Maintaining higher entropy enhances policy optimization.

“Together with Table 2, Critique-GRPO's superior performance demonstrates that maintaining higher entropy enhances policy optimization.”

In standard generation with binary rewards, the Eluder dimension scales exponentially as O(|S|^L).

“The Eluder dimension (Russo & Van Roy, 2013) scales exponentially: dim E (F) ≈ O(|S| L ).”

• cites Russo & Van Roy, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The complexity analysis assumes a reasoning problem has a hidden optimal solution sequence of L steps, each drawn from a set S.
• AssumptionThe authors state this as an assumption: The action set is all sequences of length up to L over S.
• AssumptionThe authors state this as an assumption: The hypothesis space is F.

The performance drop is likely due to greater difficulty in extracting critical information from longer contexts.

“This decline is likely due to the increased difficulty of extracting critical information from longer contexts.”

The results confirm robustness across architectures and scales.

“confirming robustness across architectures and scales”

The later entropy decrease reflects rapid internalization of the refinements.

“The subsequent entropy decrease reflects rapid internalization of these refinements.”

Amplifying rare but valuable reasoning patterns with low initial probability and high success rates is crucial for RL fine-tuning.

“The substantial gain from policy shaping demonstrates that amplifying rare but valuable reasoning patterns-those with low initial probability but high success rates-is crucial for RL fine-tuning.”

If the critique gives the correction suggestion for the first error, complexity becomes independent of the search space, O(L).

“If the critique provides the correction suggestion for the first error s * t , complexity becomes independent of the search space O(L).”

• AssumptionThe authors state this as an assumption: The complexity analysis assumes a reasoning problem has a hidden optimal solution sequence of L steps, each drawn from a set S.
• AssumptionThe authors state this as an assumption: The action set is all sequences of length up to L over S.
• AssumptionThe authors state this as an assumption: The hypothesis space is F.

The limited effectiveness of spontaneous self-reflection worsens these challenges.

“Furthermore, the limited effectiveness of spontaneous self-reflection exacerbates these challenges.”

The shaping term γ reweights gradients to give higher importance to tokens in refined responses that are currently low-probability under πθ, helping learn from valid but unfamiliar refinements.

“The shaping term γ reweights gradients to assign higher importance to tokens in the refined responses that are currently low-probability under π θ , facilitating efficient learning from valid but unfamiliar refinements.”

• AssumptionRests on an assumption: Significant distribution shifts caused by refinements produce entropy explosion and performance degradation.

Spontaneous self-reflection has limited effectiveness.

“These observations suggest the limited effectiveness of spontaneous self-reflection.”

Relying on spontaneous self-reflection for self-correction to improve problem-solving success is unreliable.

“This underscores the unreliability of relying on spontaneous self-reflection for self-correction to improve problem-solving success.”

The results point to a fundamental scaling bottleneck in scalar-reward RL, where more compute and data do not drive further reasoning gains.

“These results underscore a fundamental scaling bottleneck in scalar-reward RL, where increased compute and data fail to drive further reasoning gains.”

A likely cause of performance plateaus and persistent failures is the sparse informational content of numerical feedback.

“A likely cause of these performance plateaus and persistent failures is the sparse informational content of numerical feedback.”

The pass@k results indicate genuine improvements.

“indicating genuine improvements”

The higher entropy indicates more consistent exploration.

“indicating more consistent exploration”

Constructive feedback that localizes the first error step decomposes the problem into L independent sub-problems of size |S|, reducing complexity to linear O(|S|L).

“(2) Constructive Feedback (c CoT ): If the critique localizes the first error step t, the problem decomposes into L independent sub-problems of size |S|, reducing complexity to linear O(|S|L).”

• AssumptionThe authors state this as an assumption: The complexity analysis assumes a reasoning problem has a hidden optimal solution sequence of L steps, each drawn from a set S.
• AssumptionThe authors state this as an assumption: The action set is all sequences of length up to L over S.
• AssumptionThe authors state this as an assumption: The hypothesis space is F.

Critiques enable valuable explorations that standard generation cannot reach.

“showing that critiques enable valuable explorations unreachable through standard generation”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Critiques distinguish hypotheses more efficiently than scalar rewards.

“By characterizing critiques as Reward-Informative Feedback (Cheng et al., 2023), we show they distinguish hypotheses more efficiently than scalar rewards.”

• cites Cheng et al., 2023 sourceNot checked — no full text of the source

The advantage of CoT critiques comes from their ability to guide more effective refinements.

“This advantage arises from CoT critiques' ability to guide more effective refinements, as demonstrated in Section 3.”

For a fixed computational budget M with |S|L much less than M much less than |S|^L, critique-guided exploration has a higher probability of success than standard generation.

“Consequently, for a fixed computational budget M where |S|L ≪ M ≪ |S| L , critique-guided exploration yields a higher probability of success”

• AssumptionThe authors state this as an assumption: The complexity analysis assumes a reasoning problem has a hidden optimal solution sequence of L steps, each drawn from a set S.
• AssumptionThe authors state this as an assumption: The action set is all sequences of length up to L over S.
• AssumptionThe authors state this as an assumption: The hypothesis space is F.

The results support integrating online joint optimization through dual numeric and language feedback.

“These results strongly support the integration of online joint optimization via dual numeric and language feedback.”

Refinements from a weaker model can significantly enhance the performance of a stronger model.

“This demonstrates that refinements from a weaker model can significantly enhance the performance of a stronger model.”

• AssumptionThe authors state this as an assumption: Qwen3-8B-Base is a weaker teacher than Qwen3-8B.

Critique-GRPO accelerates convergence by altering the trajectory distribution.

“We prove that while standard GRPO converges given sufficient samples, Critique-GRPO accelerates this process by altering the trajectory distribution.”

The gradient of the shaped objective is a re-weighted version of the standard policy gradient.

“∇ θ ρ t (θ) = γπ t (π t + γ) 2 Ψ(πt) •∇ θ log π t .(23)”

Even when reward variance is positive, the update magnitude toward the optimal policy is proportional to p*.

“Even when σ R > 0, the magnitude of the update in the direction of the optimal policy is proportional to p * .”

• AssumptionRests on an assumption: In complex reasoning tasks, the probability of generating a correct solution by random sampling is small but non-negligible.

Optimizing the refinement update inherently requires reducing the probability of the initial response.

“Optimizing ∆θ ref inherently requires reducing the probability of y init .”

For group size G, the expected number of positive rewards in a standard GRPO batch is G times p*.

“For a group size G, the expected number of positive rewards in a standard GRPO batch is E[N + ] = Gp * .”

• AssumptionRests on an assumption: In complex reasoning tasks, the probability of generating a correct solution by random sampling is small but non-negligible.

If a critique also gives the correction for the first error, complexity becomes independent of vocabulary size and scales as approximately O(L).

“Furthermore, if the critique provides the correction suggestion for the first error s * t , the complexity becomes independent of the vocabulary size, scaling as dim T E (H, ℓ, ϵ) ≈ O(L).”

• AssumptionThe authors state this as an assumption: Exploration follows a uniform strategy over the effective search space.

With binary rewards, the Eluder dimension scales exponentially with sequence length, approximately O(|S|^L).

“dim E (R, ϵ) ≈ O(|S| L ).(10)”

• AssumptionThe authors state this as an assumption: Exploration follows a uniform strategy over the effective search space.

If a critique localizes the first error at step t, the problem decomposes into L sequential sub-problems each of size |S|, reducing complexity from exponential to linear, approximately O(L|S|).

“If the critique localizes the first error at step t, the problem decomposes into L sequential sub-problems, each of size |S|.This reduces the complexity from exponential to linear, i.e., dim T E (H, ℓ, ϵ) ≈ O(L|S|).”

• AssumptionThe authors state this as an assumption: Exploration follows a uniform strategy over the effective search space.

Learning from critiques is no harder than learning from rewards.

“Proposition F.5 guarantees that learning from critiques is no harder than learning from rewards.”

The reward-only binary indicator problem is equivalent to a standard bandit problem with |S|^L arms.

“The binary indicator problem is equivalent to a standard bandit problem with |S| L arms.”

The shorter responses on Qwen3-8B may result from correcting its habit of redundant and ineffective self-reflection.

“This trend can be attributed to the correction of Qwen3-8B's tendency toward redundant and ineffective self-reflection, as discussed in Appendix J.”

Setting gamma around 0.1 maximizes gradients for correction tokens that the current policy considers unlikely but plausible.

“By setting γ ≈ 0.1, we explicitly maximize gradients for”

Because P_crit is much greater than p*, the policy improvement lower bound for Critique-GRPO is much greater than that for standard GRPO.

“Since P crit ≫ p * , it follows that ∆J crit ≫ ∆J std .”

• AssumptionRests on an assumption: In complex reasoning tasks, the probability of generating a correct solution by random sampling is small but non-negligible.

Self-critique can enable self-improvement without external supervision.

“Additionally, the unsupervised approach-Critique-GRPO (self-critique & self-evaluation)-achieves an average pass@1 improvement of 2.3% over R1-GRPO, highlighting the potential of leveraging self-critique for self-improvement without any external supervision.”

For reward-informative LLF problems with constant C_F, the Transfer Eluder Dimension is bounded by the standard Eluder Dimension of the effective reward class.

“For reward-informative LLF problems with constant C F , the Transfer Eluder Dimension is bounded by the standard Eluder Dimension of the effective reward class R H = {r η : η ∈ H}:”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: A verifier exists that defines a loss measuring alignment between a hypothesis and feedback on an action, with zero loss for consistency and non-zero penalty for inconsistency.
• AssumptionThe authors state this as an assumption: Feedback is unbiased, meaning each hypothesis minimizes the expected verifier loss under its induced distribution.

If the critique also provides the correction for the first error, the dimension is further reduced to approximately O(L).

“If the critique also provides the correction suggestion for the first error s * t , the sequence of error indices in an independent action sequence becomes strictly monotonic, further reducing the dimension to d corr ≈ O(L).”

• AssumptionThe authors state this as an assumption: Exploration follows a uniform strategy over the effective search space.

The gradient update from refinements is often orthogonal to the update from initial responses.

“The gradient update from refinements, ∆θ ref , is often orthogonal to the update from initial responses, ∆θ init .”

The shaping function has peak efficiency at token probability equal to gamma.

“Peak Efficiency in the Learning Zone: Solving dΨ dπ = 0 yields a maximum at π t = γ.”

To keep the updated policy within a trust region, the influence of refined samples must be bounded by the curvature of the on-policy manifold.

“To ensure the updated policy π θ ′ remains within a trust region D KL (π θ ′ ||π θ ) ≤ δ, the influence of refined samples must be bounded by the curvature of the on-policy manifold.”

How good the exploration signals are matters more than how much exploration occurs.

“This suggests that the quality of exploration signals is more critical than the extent of exploration (as reflected solely by entropy).”

Without regularization, accumulated refinement updates drag the policy off the old manifold and lead to entropy collapse.

“Without regularization, the accumulation of ∆θ ref updates drags the policy off the manifold M old , leading to entropy collapse.”

Maintaining a certain level of entropy is important for better performance.

“Combined with the results in Table 2, the superior performance of Critique-GRPO over R1-Dr.GRPO and R1-GRPO highlights the importance of maintaining a certain level of entropy for better performance.”

If p* is less than 1/G, many batches contain no positive signal, causing wasted computation steps with zero parameter update.

“If p * < 1/G, many batches will contain no positive signal (σ R = 0), resulting in wasted computation steps where ∆θ = 0.”

• AssumptionRests on an assumption: In complex reasoning tasks, the probability of generating a correct solution by random sampling is small but non-negligible.

The probability of success is approximately 1 minus exp(-M/d).

“The probability of success is approximately: P (success) ≈ 1 -exp - M d .(14)”

For critique-guided constructive feedback, the effective dimension is the sum of L sub-problems, giving d_crit = L|S|.

“The effective dimension is the sum of these sub-problems: d crit = L t=1 |S| = L|S|. (13”

• AssumptionThe authors state this as an assumption: Exploration follows a uniform strategy over the effective search space.

Lower ratios cause performance degradation through sudden entropy-loss increases from distribution shifts introduced by refinements in later training.

“Lower ratios lead to performance degradation due to sudden increases in entropy loss, caused by distribution shifts introduced by the refinements, during later training stages.”

For a fixed computational budget M with L|S| much less than M and M much less than |S|^L, critique-guided exploration yields a significantly higher probability of success.

“Consequently, for a fixed computational budget M where L|S| ≪ M ≪ |S| L , critique-guided exploration yields a significantly higher probability of success:”

• AssumptionThe authors state this as an assumption: Exploration follows a uniform strategy over the effective search space.

The effectiveness of CoT critiques is attributable to their richness, including step-by-step evaluation and possible targeted guidance.

“The effectiveness of CoT critiques can be attributed to their richness: by providing a step-by-step evaluation of the reasoning (potentially along with targeted guidance on the correct solution approach).”

The lack of a substantial difference between indicative critiques and critiques with ground truth suggests that giving only the ground-truth answer, without explanation or reasoning, provides little additional guidance.

“However, the lack of a substantial difference between indicative critiques and critiques with ground truth suggests that providing only the ground-truth answer, without any explanation or reasoning, provides little additional guidance to the model.”

Critique-GRPO enables self-improvement via self-critiquing.

“Critique-GRPO facilitates self-improvement via self-critiquing, obtaining 66.7% pass@1 on AIME 2024 and consistent gains across pass@k scores (k=1-256) over GRPO.”

The extra computational cost of Critique-GRPO is small relative to its performance gains, making the cost justified.

“These findings highlight the substantial gains in data efficiency and model performance, demonstrating that the additional computational cost is highly justified.”

The advantage of CoT critiques likely arises because their detailed step-by-step evaluations and concise suggestions help models locate and fix errors in initial responses.

“This advantage likely stems from their detailed step-by-step evaluations and concise improvement suggestions, which help models identify and correct errors in initial responses.”

Critique-GRPO improves online reinforcement learning by letting the model learn from initial responses and self-refinements via natural language critiques, not just numerical scalar rewards.

“Critique-GRPO enhances online reinforcement learning by enabling the model to learn from both initial responses and self-refinements through natural language feedback (critiques), highlighted in green, instead of relying solely on numerical feedback (scalar rewards).”

The tendency to produce conclusive answers rather than step-by-step reasoning limits how useful expert demonstrations are as critiques.

“This behavior limits the effectiveness of expert demonstrations as critiques.”

Critique-GRPO incorporates real-time natural language feedback directly within the online reinforcement learning loop.

“In contrast, Critique-GRPO integrates real-time NLF directly into the online RL loop.”

• AssumptionRests on an assumption: PPO and GRPO perform comparably as reinforcement learning algorithms.

Combining textual feedback with online exploration for policy optimization improves flexibility and scalability.

“This combination of textual feedback with online exploration for policy optimization enhances flexibility and scalability.”

Critique-GRPO lets models refine their outputs by integrating different kinds of textual feedback, helping them identify and address potential errors.

“In contrast, our approach enables models to refine their outputs through the integration of various textual feedback, which helps identify and address potential errors.”