Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs

May combine two claims.

CriticBench evaluates critique ability using discrimination results but struggles with task-specific fine-grained metrics and reliance on costly human annotations or potentially biased GPT-4 outputs.

“CriticBench [34] evaluates critique ability using discrimination results, but it struggles with task-specific fine-grained metrics and reliance on costly human annotations or potentially biased GPT-4 outputs.”

• cites [34] [34] Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench: Benchmarking llms for critique-correct reasoning. In Annual Meeting of the Association for Computational Linguistics , 2024. - sourceSupported

May combine two claims.

DeepSeek-V3 integrates advanced reflection modules and comprehensive error-detection routines distilled from its earlier R1 series, enabling thorough multi-step reasoning at the outset.

“In particular, DeepSeek-V3 integrates advanced reflection modules and comprehensive error-detection routines distilled from its earlier R1 series, enabling thorough multi-step reasoning at the outset.”

• cites [12] [12] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, et al. Deepseek-v3 technical report. 2024. - sourcePartly supported

  The ‘Post-Training: Knowledge Distillation from DeepSeek-R1’ section says DeepSeek-V3 incorporates R1’s verification and reflection patterns and improves its reasoning performance. Section 5.1 describes training responses to use those patterns, but neither passage establishes that V3 has distinct reflection modules or comprehensive error-detection routines, or that it performs thorough multi-step reasoning ‘at the outset.’ Page numbers are unavailable in our copy. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Mixing different self-correction strategies yields further improvements in accuracy, though it reduces efficiency or incurs higher computational costs.

“2) Mixing different self-correction strategies yields further improvements, though it reduces efficiency;”

• Also stated in: § 1 Introduction · p. 2 · plain

  “Meanwhile, mixing multiple methods, while improving accuracy, incurs higher computational costs and reduced efficiency.”

• Also stated in: § 4.7 Results of Response Time · p. 8 · plain

  “Mixing self-correction methods typically results in accuracy improvements, though at the cost of increased computational overhead.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Reasoning LLMs have limited optimization under additional self-correction methods and have high time costs.

“3) Reasoning LLMs have limited optimization under additional self-correction methods and have high time costs.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “For reasoning LLMs, these methods offer only marginal gains with increased time usage.”

• Also stated in: § 4.8 Results of Reasoning LLMs · p. 8 · plain

  “Consequently, attempts to augment DeepSeek-V3 with further self-correction methods produce minimal net gains while incurring additional computational overhead.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Among the evaluated LLMs, GPT-4o and Qwen2.5-72B-Instruct consistently demonstrate superior performance as the best open-source LLM and closed-source LLM, respectively, across most methods.

“Among the evaluated LLMs, GPT-4o and Qwen2.5-72B-Instrcut consistently demonstrate superior performance as the best open-source LLM and closed-source LLM, respectively, across most methods.”

May combine two claims.

For the CoT method, most models surpass the baseline, though some fall short.

“Most models surpass the baseline ('y=0' for Base), though some fall short.”

May combine two claims.

RARR offers a balanced trade-off between efficiency and accuracy, with only 533 tokens and 2 API calls.

“As shown in Table 4, RARR offers a balanced trade-off between efficiency and accuracy, with only 533 tokens and 2 API calls.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Reflexion-v2 and RATT achieve the highest accuracies, reflecting the benefit of external retrieval or code execution, while their overhead remains manageable (below 15% additional search tokens).

“Reflexion-v2 and RATT achieve the highest accuracies, reflecting the benefit of external retrieval or code execution, while their overhead remains manageable (below 15% additional search tokens).”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B Base achieves 53.88 on AQUA.

“| LLaMA3.1-8B | Base | 75.8 | 76.16 | 53.88 | 81.55 | 74.37 | 9.74 | 73.44 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B Base achieves 76.16 on CS-QA.

“| LLaMA3.1-8B | Base | 75.8 | 76.16 | 53.88 | 81.55 | 74.37 | 9.74 | 73.44 |”

May combine two claims.

Closed-source LLMs exhibit uneven performance gains under S1; for instance, LLaMA3.1-8B-Instruct shows significant declines on AQUA and HumanEval, whereas Qwen2.5-7B-Instruct demonstrates modest gains.

“Closed-source LLMs exhibit uneven performance gains. For instance, LLaMA3.1-8B-Instruct shows significant declines on AQUA and HumanEval, whereas Qwen2.5-7B-Instruct demonstrates modest gains.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Both S1 and S2 enable significant performance gains, particularly for complex tasks requiring multi-step reasoning or domain-specific knowledge.

“Both S1 and S2 enable significant performance gains, particularly for complex tasks requiring multi-step reasoning or domain-specific knowledge.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Reasoning LLMs (e.g., DeepSeek-V3) exhibit the best overall performance, excelling particularly in tasks requiring complex reasoning capabilities, as demonstrated by their consistent dominance across all datasets and methods.

“Furthermore, reasoning LLMs (e.g., DeepSeek-V3) exhibit the best overall performance, excelling particularly in tasks requiring complex reasoning capabilities, as demonstrated by their consistent dominance across all datasets and methods.”

• AssumptionRests on an assumption: The HotpotQA, GSM8K, and GPQA datasets represent commonsense reasoning, mathematical reasoning, and complex reasoning, respectively.
• AssumptionRests on an assumption: The evaluated reasoning LLMs are representative of reasoning LLMs generally.

May combine two claims.

For the CoVe method, the majority of data points surpass the baseline, with substantial improvement magnitudes.

“The majority of data points surpass the baseline, with substantial improvement magnitudes, underscoring the effectiveness of the CoVe method.”

May combine two claims.

Reflexion-v2 increases its effectiveness, yielding improvements of +7.33% on HotpotQA and +7.00% on AQUA.

“However, Reflexion-v2 6 increases its effectiveness, yielding improvements of +7.33% on HotpotQA and +7.00% on AQUA.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Reflexion-v2 with tools achieves performance improvements, with the majority of data points surpassing the baseline.

“In contrast to the results without tools, Figure 14 highlights the performance improvements achieved by Reflexion-v2 with tools. The majority of data points surpass the baseline”

May combine two claims.

Attempts to augment DeepSeek-V3 with further self-correction methods produce minimal net gains while incurring additional computational overhead.

“Consequently, attempts to augment DeepSeek-V3 with further self-correction methods produce minimal net gains while incurring additional computational overhead.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Although S1 improves accuracy overall, the degree of improvement varies across instruction-based and reasoning LLMs.

“Although S1 improves accuracy overall, the degree of improvement varies across instruction-based and reasoning LLMs.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

The quote only partly states this claim.

Nearly all data points for RARR exceed the baseline.

“Nearly all data points exceed the baseline.”

May combine two claims.

GPT-4o shows substantial gains on GPQA, and Claude3.5 achieves similar enhancements.

“Notably, GPT-4o shows substantial gains on GPQA, whereas Claude3.5 achieves similar enhancements, highlighting the robust instruction-following adaptability of open-source architectures.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

SuperCorrect, fine-tuned on Qwen2.5-Math-7B-Instruct, demonstrates marked improvements in mathematical reasoning (e.g., on GSM8K and MATH), outperforming other methods by a wide margin.

“For example, SuperCorrect, fine-tuned on Qwen2.5-Math-7B-Instruct, demonstrates marked improvements in mathematical reasoning (e.g., on GSM8K and MATH), outperforming other methods by a wide margin.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Both intrinsic and external methods extend inference times relative to baseline approaches (Base and CoT), largely due to increased reasoning complexity or reliance on external services.

“In general, both intrinsic and external methods extend inference times relative to baseline approaches (Base and CoT), largely due to increased reasoning complexity or reliance on external services (especially for RATT).”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

DeepSeek-V3's performance remains close to the baseline across most tasks under S1; it demonstrates slight improvements on CS-QA, GPQA, and MATH, and marginal declines on others.

“DeepSeek-V3's performance remains close to the baseline across most tasks. While it demonstrates slight improvements on datasets such as CS-QA, GPQA, and MATH, it exhibits marginal declines on others.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

Both intrinsic correction and external correction outperform the baseline on most models and tasks.

“It can be observed that both intrinsic correction and external correction outperform the baseline on most models and tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA 3.1-8B-Instruct performs poorly overall.

“For instance, LLaMA 3.1-8B-Instruct performs poorly overall”

May combine two claims.

Both CoVe and RARR achieve high correction rates and low misjudgment rates, showing that self-correction methods can effectively correct wrong examples with less misjudgment.

“The results shown in Table 3 reflects that both methods achieve high CRs and low MRs, which shows that self-correction methods can effectively correct the wrong examples with less misjudgment.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Reflexion-v1 decreased by 10.90 on the third metric relative to its base.

“| LLaMA3.1-8B | Reflexion-v1 | 71.04 (-4.76) | 51.63 (-24.53) | 42.98 (-10.90) | 58.91 (-22.64) | 69.99 (-4.38) | 6.34 (-3.40) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 81.28 on the first metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe increased by 7.88 on the first metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI decreased by 14.05 on the second metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Reflexion-v1 decreased by 11.75 on the first metric relative to its base.

“| LLaMA3.1-70B | Reflexion-v1 | 69.53 (-11.75) | 66.99 (-14.89) | 31.48 (-31.17) | 63.94 (-26.69) | 65.51 (-12.70) | 16.94 (+1.32) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 62.65 on the third metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Self-Refine increased by 3.34 on the third metric relative to its base.

“| LLaMA3.1-70B | Self-Refine | 84.97 (+3.69) | 83.31 (+1.43) | 65.99 (+3.34) | 91.76 (+1.13) | 81.88 (+3.67) | 41.08 (+25.46) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 78.21 on the fifth metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Self-Refine increased by 10.88 on the sixth metric relative to its base.

“| LLaMA3.1-8B | Self-Refine | 61.59 (-14.21) | 63.91 (-12.25) | 52.39 (-1.49) | 76.41 (-5.14) | 82.11 (+7.74) | 20.62 (+10.88) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe increased by 25.28 on the seventh metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT increased by 5.29 on the fourth metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT decreased by 15.87 on the seventh metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe decreased by 5.21 on the second metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT increased by 0.96 on the second metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI decreased by 3.60 on the first metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI decreased by 1.39 on the fifth metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Reflexion-v1 decreased by 3.40 on the sixth metric relative to its base.

“| LLaMA3.1-8B | Reflexion-v1 | 71.04 (-4.76) | 51.63 (-24.53) | 42.98 (-10.90) | 58.91 (-22.64) | 69.99 (-4.38) | 6.34 (-3.40) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with CoT achieves 78.28 on GSM8K.

“| LLaMA3.1-8B | CoT | 72.14 (-3.66) | 73.85 (-2.31) | 53.88 (0) | 78.28 (-3.27) | 77.65 (+3.28) | 13.11 (+3.37) | 51.83 (-21.61) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Self-Refine increased by 7.74 on the fifth metric relative to its base.

“| LLaMA3.1-8B | Self-Refine | 61.59 (-14.21) | 63.91 (-12.25) | 52.39 (-1.49) | 76.41 (-5.14) | 82.11 (+7.74) | 20.62 (+10.88) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Self-Refine increased by 3.67 on the fifth metric relative to its base.

“| LLaMA3.1-70B | Self-Refine | 84.97 (+3.69) | 83.31 (+1.43) | 65.99 (+3.34) | 91.76 (+1.13) | 81.88 (+3.67) | 41.08 (+25.46) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 47.5 on the third metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT decreased by 15.79 on the seventh metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe increased by 23.46 on the sixth metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 15.62 on the sixth metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Self-Refine decreased by 5.14 on the fourth metric relative to its base.

“| LLaMA3.1-8B | Self-Refine | 61.59 (-14.21) | 63.91 (-12.25) | 52.39 (-1.49) | 76.41 (-5.14) | 82.11 (+7.74) | 20.62 (+10.88) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI decreased by 4.66 on the seventh metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT increased by 8.12 on the sixth metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT decreased by 1.98 on the first metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 90.63 on the fourth metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 90.23 on the fourth metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI increased by 10.27 on the sixth metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Reflexion-v1 decreased by 4.76 on the first metric relative to its base.

“| LLaMA3.1-8B | Reflexion-v1 | 71.04 (-4.76) | 51.63 (-24.53) | 42.98 (-10.90) | 58.91 (-22.64) | 69.99 (-4.38) | 6.34 (-3.40) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 79.11 on the seventh metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with CoVe achieves 50.66 on AQUA.

“| LLaMA3.1-8B | CoVe | 67.42 (-8.38) | 65.61 (-10.55) | 50.66 (-3.22) | 80.52 (-1.03) | 83.85 (+9.48) | 15.36 (+5.62) | 63.22 (-10.22) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Reflexion-v1 decreased by 22.64 on the fourth metric relative to its base.

“| LLaMA3.1-8B | Reflexion-v1 | 71.04 (-4.76) | 51.63 (-24.53) | 42.98 (-10.90) | 58.91 (-22.64) | 69.99 (-4.38) | 6.34 (-3.40) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT decreased by 18.10 on the fifth metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe increased by 8.96 on the fifth metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe increased by 13.89 on the third metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Reflexion-v1 decreased by 12.70 on the fifth metric relative to its base.

“| LLaMA3.1-70B | Reflexion-v1 | 69.53 (-11.75) | 66.99 (-14.89) | 31.48 (-31.17) | 63.94 (-26.69) | 65.51 (-12.70) | 16.94 (+1.32) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 81.88 on the second metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT decreased by 18.56 on the second metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 7.53 on the sixth metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Self-Refine increased by 1.43 on the second metric relative to its base.

“| LLaMA3.1-70B | Self-Refine | 84.97 (+3.69) | 83.31 (+1.43) | 65.99 (+3.34) | 91.76 (+1.13) | 81.88 (+3.67) | 41.08 (+25.46) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Reflexion-v1 decreased by 24.53 on the second metric relative to its base.

“| LLaMA3.1-8B | Reflexion-v1 | 71.04 (-4.76) | 51.63 (-24.53) | 42.98 (-10.90) | 58.91 (-22.64) | 69.99 (-4.38) | 6.34 (-3.40) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Reflexion-v1 decreased by 14.89 on the second metric relative to its base.

“| LLaMA3.1-70B | Reflexion-v1 | 69.53 (-11.75) | 66.99 (-14.89) | 31.48 (-31.17) | 63.94 (-26.69) | 65.51 (-12.70) | 16.94 (+1.32) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

CoVe decreases LLaMA3.1-8B AQUA performance by 3.22 points relative to Base.

“| LLaMA3.1-8B | CoVe | 67.42 (-8.38) | 65.61 (-10.55) | 50.66 (-3.22) | 80.52 (-1.03) | 83.85 (+9.48) | 15.36 (+5.62) | 63.22 (-10.22) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Self-Refine decreased by 14.21 on the first metric relative to its base.

“| LLaMA3.1-8B | Self-Refine | 61.59 (-14.21) | 63.91 (-12.25) | 52.39 (-1.49) | 76.41 (-5.14) | 82.11 (+7.74) | 20.62 (+10.88) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT decreased by 5.67 on the fifth metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Self-Refine decreased by 12.25 on the second metric relative to its base.

“| LLaMA3.1-8B | Self-Refine | 61.59 (-14.21) | 63.91 (-12.25) | 52.39 (-1.49) | 76.41 (-5.14) | 82.11 (+7.74) | 20.62 (+10.88) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT increased by 0.28 on the third metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with CoT achieves 53.88 on AQUA.

“| LLaMA3.1-8B | CoT | 72.14 (-3.66) | 73.85 (-2.31) | 53.88 (0) | 78.28 (-3.27) | 77.65 (+3.28) | 13.11 (+3.37) | 51.83 (-21.61) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoT increased by 4.85 on the first metric relative to its base.

“| LLaMA3.1-70B | CoT | 86.13 (+4.85) | 82.84 (+0.96) | 62.93 (+0.28) | 95.92 (+5.29) | 60.11 (-18.10) | 23.74 (+8.12) | 46.39 (-15.79) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Reflexion-v1 decreased by 31.17 on the third metric relative to its base.

“| LLaMA3.1-70B | Reflexion-v1 | 69.53 (-11.75) | 66.99 (-14.89) | 31.48 (-31.17) | 63.94 (-26.69) | 65.51 (-12.70) | 16.94 (+1.32) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with CoVe decreased by 4.13 on the fourth metric relative to its base.

“| LLaMA3.1-70B | CoVe | 89.16 (+7.88) | 76.67 (-5.21) | 76.54 (+13.89) | 86.5 (-4.13) | 87.17 (+8.96) | 39.08 (+23.46) | 87.46 (+25.28) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Self-Refine increased by 1.13 on the fourth metric relative to its base.

“| LLaMA3.1-70B | Self-Refine | 84.97 (+3.69) | 83.31 (+1.43) | 65.99 (+3.34) | 91.76 (+1.13) | 81.88 (+3.67) | 41.08 (+25.46) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Reflexion-v1 decreased by 26.69 on the fourth metric relative to its base.

“| LLaMA3.1-70B | Reflexion-v1 | 69.53 (-11.75) | 66.99 (-14.89) | 31.48 (-31.17) | 63.94 (-26.69) | 65.51 (-12.70) | 16.94 (+1.32) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Self-Refine increased by 25.46 on the sixth metric relative to its base.

“| LLaMA3.1-70B | Self-Refine | 84.97 (+3.69) | 83.31 (+1.43) | 65.99 (+3.34) | 91.76 (+1.13) | 81.88 (+3.67) | 41.08 (+25.46) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI decreased by 1.43 on the third metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B Base scored 62.18 on the seventh metric.

“| LLaMA3.1-70B | Base | 81.28 | 81.88 | 62.65 | 90.63 | 78.21 | 15.62 | 62.18 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 74.75 on the second metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Reflexion-v1 decreased by 4.38 on the fifth metric relative to its base.

“| LLaMA3.1-8B | Reflexion-v1 | 71.04 (-4.76) | 51.63 (-24.53) | 42.98 (-10.90) | 58.91 (-22.64) | 69.99 (-4.38) | 6.34 (-3.40) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

CoT does not change LLaMA3.1-8B AQUA performance relative to Base (0 change).

“| LLaMA3.1-8B | CoT | 72.14 (-3.66) | 73.85 (-2.31) | 53.88 (0) | 78.28 (-3.27) | 77.65 (+3.28) | 13.11 (+3.37) | 51.83 (-21.61) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with RCI decreased by 3.52 on the fourth metric relative to its base.

“| LLaMA3.1-70B | RCI | 77.68 (-3.60) | 67.83 (-14.05) | 61.22 (-1.43) | 87.11 (-3.52) | 76.63 (-1.39) | 25.89 (+10.27) | 57.52 (-4.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT increased by 4.64 on the sixth metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT increased by 3.25 on the third metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 74.28 on the fifth metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B Base scored 74.05 on the first metric.

“| Qwen2.5-7B | Base | 74.05 | 74.75 | 47.5 | 90.23 | 74.28 | 7.53 | 79.11 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoT decreased by 3.22 on the fourth metric relative to its base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with CoVe achieves 15.36 on GPQA.

“| LLaMA3.1-8B | CoVe | 67.42 (-8.38) | 65.61 (-10.55) | 50.66 (-3.22) | 80.52 (-1.03) | 83.85 (+9.48) | 15.36 (+5.62) | 63.22 (-10.22) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-70B with Reflexion-v1 increased by 1.32 on the sixth metric relative to its base.

“| LLaMA3.1-70B | Reflexion-v1 | 69.53 (-11.75) | 66.99 (-14.89) | 31.48 (-31.17) | 63.94 (-26.69) | 65.51 (-12.70) | 16.94 (+1.32) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLaMA3.1-8B with Self-Refine decreased by 1.49 on the third metric relative to its base.

“| LLaMA3.1-8B | Self-Refine | 61.59 (-14.21) | 63.91 (-12.25) | 52.39 (-1.49) | 76.41 (-5.14) | 82.11 (+7.74) | 20.62 (+10.88) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI decreased by 3.32 on the fifth metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

May combine two claims.

Qwen2.5-72B with CoVe achieves a score of 75.17 on the third benchmark column, an improvement of 17.59 over its base.

“75.17 (+17.59)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For GPT-3.5, CoVe achieved a score of 81.49 on the third benchmark, an increase of 26.34 points.

“| GPT-3.5 | CoVe | 85.17 (+2.23) | 89.67 (+11.75) | 81.49 (+26.34) | 94.98 (+15.84) | 83.75 (+13.31) | 48.5 (+21.21) | 83.95 (+3.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI increased by 7.05 on the sixth metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Qwen2.5-72B with Reflexion-v1 achieves a score of 69.65 on the first benchmark column, a decrease of 13.98 from its base.

“69.65 (-13.98)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI increased by 7.71 on the third metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI increased by 4.13 on the fourth metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe increased by 6.18 on the seventh metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For GPT-3.5, Reflexion-v1 achieved a score of 71.88 on the first benchmark, a decrease of 11.06 points.

“| GPT-3.5 | Reflexion-v1 | 71.88 (-11.06) | 62.11 (-15.81) | 44.76 (-10.39) | 72.74 (-6.4) | 59.12 (-11.32) | 21.85 (-5.44) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI decreased by 7.80 on the first metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen2.5-7B with RCI achieves a score of 84.63 on the first benchmark column, an improvement of 1.00 over its base.

“84.63 (+1.00)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For GPT-3.5, CoVe achieved a score of 89.67 on the second benchmark, an increase of 11.75 points.

“| GPT-3.5 | CoVe | 85.17 (+2.23) | 89.67 (+11.75) | 81.49 (+26.34) | 94.98 (+15.84) | 83.75 (+13.31) | 48.5 (+21.21) | 83.95 (+3.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe decreased by 0.80 on the first metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen2.5-7B with Self-Refine achieves a score of 75.46 on the first benchmark column, an improvement of 1.41 over its base.

“75.46 (+1.41)”

May combine two claims.

For GPT-3.5, Self-Refine achieved a score of 91.47 on the first benchmark, an increase of 8.53 points.

“| GPT-3.5 | Self-Refine | 91.47 (+8.53) | 82.64 (+4.72) | 69.8 (+14.64) | 98.18 (+19.04) | 85.34 (+14.9) | 52.14 (+24.85) | - |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For GPT-3.5, CoVe achieved a score of 85.17 on the first benchmark, an increase of 2.23 points.

“| GPT-3.5 | CoVe | 85.17 (+2.23) | 89.67 (+11.75) | 81.49 (+26.34) | 94.98 (+15.84) | 83.75 (+13.31) | 48.5 (+21.21) | 83.95 (+3.66) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Qwen2.5-72B with Self-Refine achieves a score of 31.19 on the sixth benchmark column, an improvement of 21.34 over its base.

“31.19 (+21.34)”

May combine two claims.

Qwen2.5-72B with CoVe achieves a score of 88.95 on the first benchmark column, an improvement of 5.32 over its base.

“88.95 (+5.32)”

May combine two claims. The quote only partly states this claim.

Qwen2.5-72B with Self-Refine achieves a score of 90.59 on the first benchmark column, an improvement of 6.96 over its base.

“90.59 (+6.96)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe increased by 11.31 on the fifth metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

May combine two claims.

Qwen2.5-7B with CoT achieves a score of 89.87 on the first benchmark column, an improvement of 6.24 over its base.

“89.87 (+6.24)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen2.5-72B with CoVe achieves a score of 63.56 on the seventh benchmark column, a decrease of 22.57 from its base.

“63.56 (-22.57)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe increased by 10.45 on the third metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen2.5-7B with Reflexion-v1 achieves a score of 60.96 on the first benchmark column, a decrease of 13.09 from its base.

“60.96 (-13.09)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI decreased by 8.14 on the second metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen2.5-7B with Reflexion-v1 achieves a score of 61.17 on the fourth benchmark column, a decrease of 29.06 from its base.

“61.17 (-29.06)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Qwen2.5-7B with Self-Refine achieves a score of 32.77 on the sixth benchmark column, an improvement of 25.24 over its base.

“32.77 (+25.24)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with RCI decreased by 21.12 on the seventh metric relative to its base.

“| Qwen2.5-7B | RCI | 66.25 (-7.80) | 66.61 (-8.14) | 55.21 (+7.71) | 94.36 (+4.13) | 70.96 (-3.32) | 14.58 (+7.05) | 57.99 (-21.12) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Claude3.5 with RCI achieves a score of 86.24 on the first benchmark column, a decrease of 2.05 from its base.

“86.24 (-2.05)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe decreased by 16.39 on the second metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe increased by 1.13 on the fourth metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

May combine two claims.

Claude3.5 with CoT achieves a score of 92.3 on the first benchmark column, an improvement of 4.01 over its base.

“92.3 (+4.01)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Claude3.5 with CoT achieves a score of 64.51 on the third benchmark column, a decrease of 16.75 from its base.

“64.51 (-16.75)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Qwen2.5-7B with CoVe increased by 28.69 on the sixth metric relative to its base.

“| Qwen2.5-7B | CoVe | 73.25 (-0.80) | 58.36 (-16.39) | 57.95 (+10.45) | 91.36 (+1.13) | 85.59 (+11.31) | 36.22 (+28.69) | 85.29 (+6.18) |”

May combine two claims.

For LLaMA3.1-8B on HotpotQA, Reflexion-v2 achieves 89.02, an improvement of 13.22 over Base.

“LLaMA3.1-8B | Reflexion-v2 | 89.02 (+13.22) | 78.66 (+2.5) | 59.77 (+5.89) | 83.9 (+2.35) | 78.14 (+3.77) | 22.14 (+12.4) | - |”

May combine two claims.

For LLaMA3.1-8B on GPQA, RARR achieves 34.58, an improvement of 24.84 over Base.

“LLaMA3.1-8B | RARR | 81.31 (+5.51) | 74.6 (-1.56) | 56.54 (+2.66) | 79.31 (-2.24) | 84.43 (+10.06) | 34.58 (+24.84) | 67.58 (-5.86) |”

May combine two claims.

For LLaMA3.1-8B on GPQA, Reflexion-v2 achieves 22.14, an improvement of 12.4 over Base.

“LLaMA3.1-8B | Reflexion-v2 | 89.02 (+13.22) | 78.66 (+2.5) | 59.77 (+5.89) | 83.9 (+2.35) | 78.14 (+3.77) | 22.14 (+12.4) | - |”

May combine two claims.

For LLaMA3.1-70B on GPQA, RARR achieves 35.79, an improvement of 20.17 over Base.

“LLaMA3.1-70B | RARR | 90.49 (+9.21) | 79.93 (-1.95) | 67.29 (+4.64) | 88.5 (-2.13) | 87.8 (+9.59) | 35.79 (+20.17) | 78.96 (+16.78) |”

May combine two claims.

For LLaMA3.1-70B on MATH, RARR achieves 87.8, an improvement of 9.59 over Base.

“LLaMA3.1-70B | RARR | 90.49 (+9.21) | 79.93 (-1.95) | 67.29 (+4.64) | 88.5 (-2.13) | 87.8 (+9.59) | 35.79 (+20.17) | 78.96 (+16.78) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Qwen2.5-7B, RARR improves performance over Base by 26.91 on the sixth benchmark (column 6 of the results table).

“| Qwen2.5-7B | RARR | 71.67 (-2.38) | 75.52 (+0.77) | 45.61 (-1.89) | 85.22 (-5.01) | 79.44 (+5.16) | 34.44 (+26.91) | 85.71 (+6.6) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Qwen2.5-7B, RATT improves performance on the third benchmark (column 3 of the results table) by 17.67 relative to Base.

“| Qwen2.5-7B | RATT | 72.4 (-1.65) | 84.55 (+9.8) | 65.17 (+17.67) | 94.92 (+4.69) | 84.61 (+10.33) | 14.61 (+7.08) | 80.69 (+1.58) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Qwen2.5-7B, CoT decreases performance on the second benchmark (column 2 of the results table) by 18.56 relative to Base.

“| Qwen2.5-7B | CoT | 72.07 (-1.98) | 56.19 (-18.56) | 50.75 (+3.25) | 87.01 (-3.22) | 68.61 (-5.67) | 12.17 (+4.64) | 63.24 (-15.87) |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For LLaMA3.1-8B on GSM8K, RATT achieves 72.34, a decrease of 9.21 from Base.

“LLaMA3.1-8B | RATT | 76.82 (+1.02) | 76.72 (+0.56) | 54.94 (+1.06) | 72.34 (-9.21) | 81.16 (+6.79) | 13.83 (+4.09) | 62.33 (-11.11) |”

May combine two claims.

For LLaMA3.1-70B on HotpotQA, RARR achieves 90.49, an improvement of 9.21 over Base.

“LLaMA3.1-70B | RARR | 90.49 (+9.21) | 79.93 (-1.95) | 67.29 (+4.64) | 88.5 (-2.13) | 87.8 (+9.59) | 35.79 (+20.17) | 78.96 (+16.78) |”

May combine two claims.

For LLaMA3.1-8B on MATH, RARR achieves 84.43, an improvement of 10.06 over Base.

“LLaMA3.1-8B | RARR | 81.31 (+5.51) | 74.6 (-1.56) | 56.54 (+2.66) | 79.31 (-2.24) | 84.43 (+10.06) | 34.58 (+24.84) | 67.58 (-5.86) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Qwen2.5-72B, RARR improves performance over Base by 30.28 on the sixth benchmark (column 6 of the results table).

“| Qwen2.5-72B | RARR RATT | 87.77 (+4.14) 84.68 (+1.05) | 86.16 (+4.24) 84.12 (+2.2) | 57.23 (-0.35) 64.41 (+6.83) | 86.4 (-4.71) 90.64 (-0.47) | 73.66 (+6.75) 77.94 (+11.03) | 40.13 (+30.28) 25.74 (+15.89) | 87.63 (+1.5) 84.72 (-1.41) |”

Probably not a substantial claim: may not pass S3 (load-bearing).

For GPT-4o, RARR improves performance on the seventh benchmark (column 7 of the results table) by 17.05 relative to Base.

“| GPT-4o | RARR | 90.89 (+1.73) | 84.87 (+4.22) | 82.13 (+16.31) | 95.34 (+4.19) | 77.65 (+8.11) 81.61 (+12.07) | 35.69 (+13.2) | 94.09 (+17.05) |”

May combine two claims.

For LLaMA3.1-70B on HumanEval, RARR achieves 78.96, an improvement of 16.78 over Base.

“LLaMA3.1-70B | RARR | 90.49 (+9.21) | 79.93 (-1.95) | 67.29 (+4.64) | 88.5 (-2.13) | 87.8 (+9.59) | 35.79 (+20.17) | 78.96 (+16.78) |”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

All datasets and code assets used are publicly available and properly cited in the main paper, with the license explicitly stated for each asset and use complying with the terms.

“We make use of publicly available datasets and code assets, all of which are properly cited in the main paper. For each asset, we explicitly state the license (e.g., MIT, Apache 2.0, CC-BY 4.0) and ensure our use complies with the terms.”

Classified as probably not a substantial claim.

95% bootstrap confidence intervals were computed for each method's accuracy across five random subsamples to validate the robustness of the trade-offs in Table 4.

“To validate the robustness of the trade-offs in Table 4, we report 95% bootstrap confidence intervals for each method's accuracy across five random subsamples.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This highlights the robust instruction-following adaptability of open-source architectures.

“Notably, GPT-4o shows substantial gains on GPQA, whereas Claude3.5 achieves similar enhancements, highlighting the robust instruction-following adaptability of open-source architectures.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

May combine two claims.

DeepSeek-V3's built-in correction mechanism delivers a strong baseline performance, and this high initial performance likely limits the impact of other correction methods.

“To further investigate this phenomenon, we conducted additional experiments (see Section 4.8), revealing that DeepSeek-V3's built-in correction mechanism delivers a strong baseline performance. This high initial performance likely limits the impact of other correction methods.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper does not introduce or release any models or datasets that pose significant risks of misuse.

“The paper does not introduce or release any models or datasets that pose significant risks of misuse.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

These discrepancies likely stem from smaller parameter sizes and weaker instruction-following capabilities.

“These discrepancies likely stem from smaller parameter sizes and weaker instruction-following capabilities.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The paper specifies all necessary experimental details, including the data splits and model configuration, to understand and interpret the results.

“We specify all necessary experimental details to understand and interpret our results. This includes the data splits and model configuration.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The lag of S3 is likely due to smaller model sizes and narrower training objectives.

“Finally, S3 generally lags behind these methods, likely due to smaller model sizes and narrower training objectives (Details in Section 4.5).”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper reports the response time cost of calling the API by different methods in Section 4.7.

“We report the response time cost of calling API by different methods in Section 4.7.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The method can help researchers better understand the correction ability of LLMs.

“On the positive side, our method can help researchers understand the correction ability of LLMs better.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paper reports quantitative results for all experiments but does not include error bars or statistical significance tests.

“While we provide quantitative results for all experiments, we do not include error bars or statistical significance tests.”

Classified as probably not a substantial claim.

LLMs are used as a core component of the methodology, specifically instruction-based LLMs and reasoning-based LLMs for evaluations.

“This work involves the use of LLMs as a core component of our methodology. Specifically, we use instruction-based LLMs and reasoning-based LLMs for evaluations.”

May combine two claims.

More complex correction strategies do not always yield superior outcomes, highlighting the critical trade-off between model accuracy and computational efficiency.

“This observation underscores that more complex correction strategies do not always yield superior outcomes, highlighting the critical trade-off between model accuracy and computational efficiency.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paper provides anonymized access to key code and data used in the experiments, with detailed instructions for reproducing the main results, including environment setup, running commands, and datasets.

“We provide anonymized access to key code and data used in the experiments, with detailed instructions for reproducing the main results, including environment setup, running commands, and datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The work does not involve crowdsourcing or research with human subjects.

“The paper does not involve crowdsourcing or research with human subjects. All experiments are performed using synthetic or publicly available machine-generated datasets.”

• Also stated in: § Answer: [NA] · p. 22 · plain

  “The paper does not involve crowdsourcing or research with human subjects, and thus IRB approval is not applicable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This is likely due to constraints posed by its smaller parameter size.

“Conversely, QWQ attains considerable improvements on most datasets except CS-QA and MATH, likely reflecting the constraints posed by its smaller parameter size.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paper introduces new assets, including collected datasets for evaluating self-correction of LLMs, and provides complete documentation alongside them covering structure, usage instructions, licensing terms, known limitations, and responsible-use guidelines.

“We introduce new assets in the paper, including the collected datasets for evaluating self-correction of LLMs. We provide complete documentation alongside these assets, including descriptions of their structure, usage instructions, licensing terms, known limitations, and guidelines for responsible use.”

Classified as probably not a substantial claim.

Each dataset is sampled to include 100 examples, selected through a randomized process.

“To ensure consistency and reproducibility, each dataset is sampled to include 100 examples, selected through a randomized process.”

May combine two claims.

External correction methods demonstrate greater stability, attributed to their reliance on authoritative external resources.

“Notably, external correction methods demonstrate greater stability, which can be attributed to their reliance on authoritative external resources.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The work does not involve private or sensitive data or potentially harmful applications.

“Our work does not involve human subjects, private or sensitive data, or potentially harmful applications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The research complies with the NeurIPS Code of Ethics and all relevant ethical guidelines.

“We have carefully reviewed the NeurIPS Code of Ethics and confirm that our research complies with all relevant ethical guidelines.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Reliance on external inputs may limit the LLM's capacity for divergent or creative reasoning, resulting in steadier but less flexible performance compared to intrinsic correction methods.

“However, this reliance on external inputs may also limit the LLM's capacity for divergent or creative reasoning, resulting in steadier but less flexible performance compared to intrinsic correction methods.”

• AssumptionRests on an assumption: The three task scenarios (commonsense reasoning, mathematical reasoning, code generation) are representative of reasoning-intensive tasks for LLMs.
• AssumptionRests on an assumption: The two categories of LLMs (instruction-based and reasoning) are representative of LLMs broadly.
• AssumptionRests on an assumption: The selected datasets within each task are representative of that task.

The quote only partly states this claim.

Reflexion-v1 decreases performance for GPT-3.5 on all seven benchmarks.

“| GPT-3.5 | Reflexion-v1 | 71.88 (-11.06) | 62.11 (-15.81) | 44.76 (-10.39) | 72.74 (-6.4) | 59.12 (-11.32) | 21.85 (-5.44) | - |”

The quote only partly states this claim.

Self-Refine improves performance for GPT-3.5 on the six benchmarks for which data are reported.

“| GPT-3.5 | Self-Refine | 91.47 (+8.53) | 82.64 (+4.72) | 69.8 (+14.64) | 98.18 (+19.04) | 85.34 (+14.9) | 52.14 (+24.85) | - |”