ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent work shows that large language models, when combined with scaling strategies such as Best-of-K sampling and iterative refinement, can achieve strong performance on PBE tasks.

“Recent work shows that large language models (LLMs), when combined with scaling strategies such as Best-ofK sampling and iterative refinement, can achieve strong performance on PBE tasks [Li and Ellis, 2024, Naik et al., 2025b].”

• cites Li and Ellis, 2024 sourceSupported
• cites Naik et al., 2025bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Compute-matched evaluations show that gains from tool-augmented LLMs are largely driven by increased inference-time compute rather than effective reuse of learned tools, with limited evidence of consistent tool reuse across instances.

“Compute-matched evaluations show that their gains are largely driven by increased inferencetime compute rather than effective reuse of learned tools, with limited evidence of consistent tool reuse across instances [Sesterhenn et al., 2025].”

• cites Sesterhenn et al., 2025 sourcePartly supported

  Section 4.1.1 reports that TROVE’s advantage on MATH appears to come primarily from repeated sampling rather than toolbox learning. Section 5.3 reports evidence that learned libraries are often invoked only once, casting doubt on reuse. The claim extends these findings to tool-augmented LLMs generally, whereas this paper’s compute-matched evaluation focuses on TROVE, chiefly on MATH. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Complexity groupings follow the binning used in the original SLR-Bench paper.

“Complexity groupings follow the binning used in the original SLR-Bench paper [Helff et al., 2025].”

• cites Helff et al., 2025 sourceSupported

May combine two claims.

LLMs can solve program synthesis tasks but remain inefficient and unreliable on hard instances requiring large combinatorial search.

“LLMs can solve program synthesis tasks but remain inefficient and unreliable on hard instances requiring large combinatorial search.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Induced solvers complement LLM search, improving PBEBench-Hard accuracy from 68.4% to 85.8%.

“They also complement LLM search, improving PBEBench-Hard accuracy from 68.4% to 85.8%”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

At cascade lengths 14-17, BoK collapses to below 47% while symbolic solvers maintain 55-70% accuracy through structured search.

“at cascade lengths 14-17, BoK collapses to below 47% while symbolic solvers maintain 55-70% accuracy through structured search.”

May combine two claims.

Removing reasoning traces (no-CoT) significantly degrades performance, especially on harder instances (e.g. PBEBench-Hard accuracy drops from 74.7% to 24.8%), indicating that final programs alone are insufficient supervision for learning robust solvers.

“Removing reasoning traces (no-CoT) significantly degrades performance, especially on harder instances (e.g. PBEBench-Hard accuracy drops from 74.7% to 24.8%), indicating that final programs alone are insufficient supervision for learning robust solvers.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The trace dataset of approximately 100 examples per benchmark is representative enough of the task distribution to induce generalizable solvers.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Solver induction ablations confirm that reasoning traces are critical: removing CoT causes large accuracy drops, especially on Hard tasks.

“Solver induction ablations confirm that reasoning traces are critical - removing CoT causes large accuracy drops especially on Hard tasks”

May combine two claims.

A direct coding-agent baseline (OpenHands) achieves 87.2% accuracy on PBEBench-Lite, outperforming individual symbolic solvers but falling below LLM scaling methods at substantially higher cost.

“A direct coding-agent baseline (OpenHands) achieves 87.2% accuracy, outperforming individual symbolic solvers but falling below LLM scaling methods at substantially higher cost.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

The QO solver reaches 74.7% and the CC solver reaches 69.7% on PBEBench-Hard.

“QO solver ('100 examples + CoT, run 2') reaches 74.7% and CC solver reaches 69.7%.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Induced symbolic solvers require no LLM calls at test time and can solve a large fraction of tasks directly, often matching or exceeding LLM search on harder instances.

“These solvers require no LLM calls at test time and can solve a large fraction of tasks directly, often matching or exceeding LLM search on harder instances.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Reducing the number of CoT examples has a smaller and less consistent effect: a 48-example CoT solver achieves performance comparable to some 100-example runs, while 12 examples is clearly insufficient.

“In contrast, reducing the number of CoT examples has a smaller and less consistent effect: a 48-example CoT solver achieves performance comparable to some 100-example runs, while 12 examples is clearly insufficient.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The trace dataset of approximately 100 examples per benchmark is representative enough of the task distribution to induce generalizable solvers.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

The All Symbolic union (6 QO solvers and CC solver) improves to 84.7% on PBEBench-Hard, exceeding BoK by 16.3pp with no LLM inference.

“The All Symbolic union (6 QO solvers and CC solver) further improves to 84.7%, exceeding BoK by 16.3pp with no LLM inference.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Performance differences across solvers induced on the same trace dataset (100-examples + CoT) can be large (e.g., 53.4%-79.2% on Lite, 51.8%-74.7% on Hard), suggesting that solver quality is highly sensitive to the algorithmic strategy discovered during induction rather than trace dataset size alone.

“Notably, performance differences across solvers induced on the same trace dataset (100-examples + CoT) can be large (e.g., 53.4%-79.2% on Lite, 51.8%-74.7% on Hard), suggesting that solver quality is highly sensitive to the algorithmic strategy discovered during induction rather than trace dataset size alone.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The trace dataset of approximately 100 examples per benchmark is representative enough of the task distribution to induce generalizable solvers.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

LLM baselines achieve strong performance with test-time scaling (BoK: 93.8%, DF: 92.4%) on PBEBench-Lite, but incur substantial token usage.

“LLM baselines achieve strong performance with test-time scaling (BoK: 93.8%, DF: 92.4%), but incur substantial token usage.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

LLM-based search performs well on short, simple programs but degrades significantly on longer, compositional transformations.

“Moreover, we observe that LLM-based search performs well on short, simple programs but degrades significantly on longer, compositional transformations.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Hybrid inference with BoK + All Symbolic reaches 85.8% on PBEBench-Hard while reducing token cost by 78%.

“Hybrid inference amplifies this advantage, with BoK + All Symbolic reaching 85.8% while reducing token cost by 78%.”

May combine two claims.

Induced symbolic solvers exceed BoK on PBEBench-Hard at zero per-task LLM cost, with QO at 74.7% and CC at 69.7%.

“whereas induced symbolic solvers exceed this at zero per-task LLM cost (QO: 74.7%, CC: 69.7%)”

May combine two claims.

Increasing the maximum cascade length from 20 to 100 yields only modest gains (e.g., 69.0% → 70.2% for CC) on the historical linguistics task, suggesting that performance is not primarily limited by search depth.

“Increasing the maximum cascade length from 20 to 100 yields only modest gains (e.g., 69.0% → 70.2% for CC), suggesting that performance is not primarily limited by search depth.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

The direct coding-agent baseline, despite consuming the most tokens of any system, achieves the lowest hard-tier performance.

“the direct coding-agent baseline, despite consuming the most tokens of any system, achieves the lowest hard-tier performance”

May combine two claims.

On SLR-Bench, BoK achieves 100% on Basic and Easy but collapses to 17.2% on Hard, while DF is more balanced (34.4% Hard) at comparable token cost.

“BoK achieves 100% on Basic and Easy but collapses to 17.2% on Hard, while DF is more balanced (34.4% Hard) at comparable token cost.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

The CC solver reaches 80.4% accuracy on PBEBench-Lite at zero inference cost, outperforming single-attempt LLM baselines.

“Induced symbolic solvers are competitive as standalone systems, with the CC solver reaching 80.4% accuracy at zero inference cost, outperforming single-attempt LLM baselines.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Hybrid methods combine both strengths: DF handles the Basic and Easy tiers near-perfectly while the symbolic solver provides hard-tier robustness that LLM search alone cannot reach, yielding the best overall and hard-tier results across all benchmarks evaluated.

“Hybrid methods combine both strengths: DF handles the Basic and Easy tiers near-perfectly while the symbolic solver provides hard-tier robustness that LLM search alone cannot reach, yielding the best overall and hard-tier results across all benchmarks evaluated.”

May combine two claims.

On SLR-Bench Hard, BoK collapses to 17.2% while the CC Solver reaches 46.8%, matching frontier models (o3, GPT-5) at zero inference cost.

“BoK collapses to 17.2% on Hard while the CC Solver reaches 46.8% - matching frontier models (o3, GPT-5) at zero inference cost.”

May combine two claims.

BoK + All Symbolic reaches 85.8% on PBEBench-Hard while reducing reported token cost by 78% relative to BoK alone (71.6M vs. 332.1M tokens).

“BoK + All Symbolic reaches 85.8% while reducing reported token cost by 78% relative to BoK alone (71.6M vs. 332.1M tokens)”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

All solvers are standalone, require no LLM calls at inference time, and are effectively deterministic (<1% per-task flips on PBEBench-Hard).

“All solvers are standalone, require no LLM calls at inference time, and are effectively deterministic ( &lt; 1% per-task flips on PBEBench-Hard).”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Combining solvers via union substantially improves robustness on the historical linguistics task, reaching up to 80.1% accuracy and demonstrating strong complementarity.

“Combining solvers via union substantially improves robustness, reaching up to 80.1% accuracy and demonstrating strong complementarity.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

The best hybrid, DF + CC + QO, achieves 86.7% overall accuracy on SLR-Bench, almost +9pp above the best SLR-Bench leaderboard result (o3: 77.8%), at 138.8M tokens ($16.19), compared to $207.24 for o3.

“The best hybrid, DF + CC + QO, achieves 86.7% overall accuracy - almost +9pp above the best SLR-Bench leaderboard result (o3: 77.8%) - at 138.8M tokens ($16.19), compared to $207.24 for o3.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

In contrast to PBEBench, all systems predict simpler rules than ground truth (∆ < 0) on SLR-Bench, reflecting the abundance of shorter equivalent Prolog forms.

“In contrast to PBEBench, all systems predict simpler rules than ground truth ( ∆ &lt; 0 ), reflecting the abundance of shorter equivalent Prolog forms”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

BoK + QO achieves Pareto efficiency on PBEBench-Lite, matching peak BoK accuracy while reducing both token usage and total cost ($9.97 vs. $12.20).

“Among hybrids, BoK + QO achieves Pareto efficiency, matching peak BoK accuracy while reducing both token usage and total cost ($9.97 vs. $12.20).”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

DF + CC Solver lifts SLR-Bench Hard accuracy to 58.0%, surpassing all prior reported baselines on this tier.

“DF + CC Solver lifts Hard accuracy to 58.0%, surpassing all prior reported baselines on this tier.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

One independently induced solver (the unique-op permutation strategy) achieves only 4% accuracy under distribution shift on the historical linguistics task, illustrating that algorithmic brittleness, not data shift alone, limits generalization.

“However, one independently induced solver (the unique-op permutation strategy in §F.2) achieves only 4% accuracy under distribution shift, illustrating that algorithmic brittleness, not data shift alone, limits generalization.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

BoK + All Symbolic lowers total tokens from 68.0M to 43.5M (-36%) on PBEBench-Lite while maintaining peak accuracy (93.9%).

“Hybrid inference substantially reduces LLM token usage: BoK + All Symbolic lowers total tokens from 68.0M to 43.5M (-36%) while maintaining peak accuracy (93.9%).”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Across all methods on PBEBench-Lite, predicted cascade complexity remains close to ground truth (∆ ≈ +2), indicating that accuracy gains are not attributable to trivial or degenerate programs.

“Across all methods, predicted cascade complexity remains close to ground truth ( ∆ ≈ +2 ), indicating that accuracy gains are not attributable to trivial or degenerate programs.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Combining multiple solvers yields 91.3% accuracy on PBEBench-Lite without any LLM usage, within 2.5pp of BoK.

“Combining multiple solvers yields 91.3% accuracy without any LLM usage, within 2.5pp of BoK.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Predicted cascade complexity substantially overshoots ground truth (∆ ≈ +5 to +8) on PBEBench-Hard, reflecting the difficulty of recovering compact solutions at long cascade lengths.

“Predicted cascade complexity substantially overshoots ground truth ( ∆ ≈ +5 -+8 ) on Hard, reflecting the difficulty of recovering compact solutions at long cascade lengths”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

BoK achieves only 68.4% on PBEBench-Hard at a cost of $57.83.

“BoK achieves only 68.4% at substantial cost ($57.83)”

May combine two claims.

The direct coding-agent baseline on SLR-Bench consumes the most tokens by far while achieving the lowest overall accuracy and the weakest Hard-tier performance of any method evaluated.

“The direct coding-agent baseline consumes the most tokens by far while achieving the lowest overall accuracy and the weakest Hardtier performance of any method we evaluate.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

On PBEBench-Hard, LLM test-time scaling becomes the weaker baseline while symbolic solvers emerge as the primary competitive systems.

“On PBEBench-Hard, the performance landscape inverts: LLM test-time scaling becomes the weaker baseline while symbolic solvers emerge as the primary competitive systems.”

May combine two claims.

The CC Solver reaches 46.8% on SLR-Bench Hard, matching the best reported frontier-model results (o3: 45%, GPT-5: 46%) at zero per-task LLM cost.

“the CC Solver reaches 46.8% on Hard - matching the best reported frontier-model results (o3: 45%, GPT-5: 46%) at zero per-task LLM cost.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Solver inference completes in under 5 minutes for all 1000+ tasks (pure Python, no LLM calls), compared to ∼2 days for DF or BoK at 8 parallel workers - a wall-clock gap of over 500×.

“Beyond tokens, solver inference completes in under 5 minutes for all 1000+ tasks (pure Python, no LLM calls), compared to ∼ 2 days for DF or BoK at 8 parallel workers - a wall-clock gap of over 500 × (Appendix C).”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

PBEBench-Lite is nearly saturated by the scaling strategies and they tend to dominate even for harder cases.

“The figures show that due to the simpler nature of the PBEBench-Lite dataset it is nearly saturated by the scaling strategies and they tend to dominate even for harder cases.”

May combine two claims.

Both BoK and DF solve cascade-length-2 tasks perfectly but collapse at length 5: BoK accuracy drops to 79.3% and DF to 74.6%.

“Both methods solve cascade-length-2 tasks perfectly but collapse at length 5: BoK accuracy drops to 79.3% and DF to 74.6%”

May combine two claims.

Tasks with no BFCC interactions are easiest (92.7%); adding feeding interactions consistently reduces accuracy more than bleeding alone.

“Tasks with no BFCC interactions are easiest (92.7%); adding feeding interactions consistently reduces accuracy more than bleeding alone.”

May combine two claims.

Run-to-run variance is large: the three 100-example CoT Qwen runs span 53.4%-79.2% on Lite and 51.8%-74.7% on Hard, with each run inventing a qualitatively different algorithm.

“Second, run-to-run variance is large: the three 100-example CoT Qwen runs span 53.4%-79.2% on Lite and 51.8%-74.7% on Hard, with each run inventing a qualitatively different algorithm.”

May combine two claims.

Despite near-zero accuracy at CL 20, mean reward remains above 0.92 at every level, confirming that failures are dominated by near-misses rather than complete failures.

“Despite near-zero accuracy at CL 20, mean reward remains above 0.92 at every level, confirming that failures are dominated by near-misses rather than complete failures.”

May combine two claims.

Both solvers are effectively deterministic: the CC solver agrees on 99.6% of tasks across runs and the QO solver on 97.0%.

“Both solvers are effectively deterministic: the CC solver agrees on 99.6% of tasks across runs and the QO solver on 97.0%.”

May combine two claims.

The no-CoT solver takes more turns than the corresponding CoT run, suggesting that reasoning traces reduce the need for unguided probing.

“Fourth, the no-CoT solver takes more turns than the corresponding CoT run, suggesting that reasoning traces reduce the need for unguided probing.”

May combine two claims.

Qwen SLR run 2 performance is weaker: the verifier budget is too tight for medium and hard tasks, and the Python approximation may reject rules the official verifier would accept.

“Despite the more elaborate design, performance is weaker: the verifier budget is too tight for medium and hard tasks, and the Python approximation may reject rules the official verifier would accept.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Run 3 uses a 'unique-op forcing' strategy that fails under real IPA distributions.

“† Run 3 uses a 'unique-op forcing' strategy that fails under real IPA distributions.”

May combine two claims.

The dominant failure mode on both PBEBench-Lite and PBEBench-Hard is near-misses: the solver finds a cascade that correctly transforms all but 1-3 input-output pairs, reflected in high mean rewards (0.944 on Lite, 0.987 on Hard) despite imperfect accuracy.

“The dominant failure mode on both PBEBench-Lite and PBEBenchHard is near-misses : the solver finds a cascade that correctly transforms all but 1-3 input-output pairs, reflected in high mean rewards (0.944 on Lite, 0.987 on Hard) despite imperfect accuracy.”

May combine two claims.

LLM reasoning traces can be compiled into reusable symbolic solvers that are strong standalone systems, match or exceed frontier-model performance on hard tasks at zero per-task inference cost, and substantially reduce token usage as hybrids, Pareto-dominating direct per-task agentic effort.

“We show that LLM reasoning traces can be compiled into reusable symbolic solvers that are strong standalone systems, match or exceed frontier-model performance on hard tasks at zero per-task inference cost, and substantially reduce token usage as hybrids - Pareto-dominating direct per-task agentic effort.”

May combine two claims.

Removing CoT from the 100-example demos drops performance from 53.4% to 42.1% on Lite and from 58.9% to 24.8% on Hard.

“Removing CoT from the 100-example demos drops performance from 53.4% to 42.1% on Lite and from 58.9% to 24.8% on Hard.”

May combine two claims.

All discrepancies between runs occur at the partial-credit boundary (0.96↔1.0 or 0.98↔1.0); no task flips between clearly solved and clearly failed.

“All discrepancies between runs occur at the partial-credit boundary ( 0 . 96 ↔ 1 . 0 or 0 . 98 ↔ 1 . 0 ); no task flips between clearly solved and clearly failed.”

May combine two claims.

The hardest BFCC category is all-four-BFCC (40.2%), which constitutes 22% of the dataset; dense mutual ordering interactions are the primary bottleneck for beam-search-based solvers.

“The hardest category is all-four-BFCC (40.2%), which constitutes 22% of the dataset; dense mutual ordering interactions are the primary bottleneck for beam-search-based solvers.”

May combine two claims.

Directly solving the problems with a coding agent performs worse compared to simple LLM scaling but is better than the solvers.

“Additionally directly solving the problems with a coding agent performs worse compared to simple LLM scaling but is better than the solvers.”

May combine two claims.

DF consumes an average of 285K tokens per task at length 5 (13 times more than at length 2), and BoK consumes 160K tokens (9 times more).

“DF consumes an average of 285K tokens per task at length 5 ( 13 × more than at length 2), and BoK consumes 160K tokens ( 9 × more).”

May combine two claims.

Across independently induced solvers, performance ranges from 4.1% to 70.5%, reflecting substantial algorithmic variability.

“Across independently induced solvers, performance ranges from 4.1% to 70.5%, reflecting substantial algorithmic variability.”

May combine two claims.

Symbolic solvers still lead to token efficiency and the performance gap is smaller when considering partial correctness via the mean reward.

“However the symbolic solvers still lead to token efficiency and the performance gap is smaller when considering partial correctness via the mean reward.”

May combine two claims.

In contrast to PBEBench-Lite, LLM test-time scaling (BoK) collapses at long cascades while the symbolic solvers maintain substantially higher accuracy through CL 17.

“In contrast to PBEBench-Lite, LLM test-time scaling (BoK) collapses at long cascades while the symbolic solvers maintain substantially higher accuracy through CL 17, illustrating the complementarity that makes hybrid inference most effective on Hard.”

May combine two claims.

Solvers tend to induce more complex programs than LLM scaling, but the coding agent due to its multi-turn refinement strategy ends up achieving the lowest complexity/simplest programs.

“For complexity the solvers tend to induce more complex programs than LLM scaling but the coding agent due to its multi-turn refinement strategy ends up achieving the lowest complexity/simplest programs.”

May combine two claims.

The Claude Code SLR solver achieves 68.4% overall accuracy, with 100% on Basic, 78.4% on Easy, and approximately 48% on Medium/Hard; its high mean score indicates failures are often near-misses rather than complete failures.

“It achieves 68.4% overall accuracy, with 100% on Basic, 78.4% on Easy, and approximately 48% on Medium/Hard; its high mean score indicates that failures are often near-misses rather than complete failures.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Each OpenHands run produces a different solver algorithm despite identical inputs, indicating exploration over algorithmic space.

“Each OpenHands run produces a different solver algorithm despite identical inputs, indicating exploration over algorithmic space.”

May combine two claims.

Symbolic solvers incur zero per-task inference cost; construction costs are small (∼$0.3-$1.3 per solver) and amortize after a handful of tasks.

“Symbolic solvers incur zero per-task inference cost; construction costs are small ( ∼ $0 . 3 -$1.3 per solver) and amortize after a handful of tasks.”

May combine two claims.

Ensembling recovers variance across runs and yields the strongest overall performance, achieving 91.3% on Lite and 84.7% on Hard without any LLM inference cost.

“Ensembling recovers variance across runs and yields the strongest overall performance, achieving 91.3% on Lite and 84.7% on Hard without any LLM inference cost.”

May combine two claims.

The token costs for each solver are in a small range of $0.3-1.34, showing the affordability of symbolic solver induction.

“The cost tends to be in a small range of $0.3-1.34 showing the affordability of symbolic solver induction.”

May combine two claims.

On PBEBench-Lite, where the CC Solver covers 80.4% of tasks perfectly, token savings are substantial (approximately 33% for BoK + CC).

“On PBEBench-Lite, where the CC Solver covers 80.4% of tasks perfectly, token savings are substantial ( ∼ 33% for BoK + CC)”

May combine two claims.

Symbolic solvers overshoot ground-truth complexity substantially (∆≈+5-+8) on PBEBench-Hard due to valid-but-verbose inductive solutions at long cascades; BoK complexity tracks GT more closely.

“Symbolic solvers overshoot ground-truth complexity substantially ( ∆ ≈ +5 -+8 ) due to valid-but-verbose inductive solutions at long cascades; BoK complexity tracks GT more closely.”

May combine two claims.

The induced symbolic solvers avoid enumeration by extracting candidate predicates directly from input-output differences, pruning replacements to a small candidate set, and applying greedy or beam search over cascades, evaluating hundreds to thousands of candidates per task rather than enumerating the full combinatorial space.

“The induced symbolic solvers avoid this enumeration by extracting candidate predicates directly from input-output differences, pruning replacements to a small candidate set, and applying greedy or beam search over cascades. Thus, they evaluate hundreds to thousands of candidates per task rather than enumerating the full combinatorial space.”

• Also stated in: § F.3 Additional Details: PBEBench-Hard · p. 23 · plain

  “The induced symbolic solvers reduce this by extracting candidate predicates directly from input-output diffs, restricting replacement candidates to a small typed set, and applying beam or greedy search so that only hundreds to thousands of candidates are evaluated per task.”

May combine two claims.

Symbolic solvers overshoot ground-truth complexity (∆≈+3) on PBEBench-Lite; the direct coding agent (QO Agent) achieves the lowest complexity through multi-turn refinement.

“Symbolic solvers overshoot ground-truth complexity ( ∆ ≈ +3 ); the direct coding agent (QO Agent) achieves the lowest complexity through multi-turn refinement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

BoK and CC Solver do not differ significantly in accuracy (∆ = -0.30, p = 0.896).

“| BoK | CC Solver | - 0.30 | 0.896 | +0.0265 | < 0 . 001 ∗∗∗ |”

May combine two claims.

BoK+QO vs. DF+QO accuracy difference is significant by McNemar (∆Acc = +7.80 pp, p<0.001) but not by Wilcoxon (p = 0.359).

“The BoK+QO vs. DF+QO comparison ( ∆ Acc = +7 . 80 pp, p&lt; 0 . 001 ) is significant by McNemar but not by Wilcoxon ( p = 0 . 359 )”

May combine two claims.

CC Solver accuracy degrades gracefully with rule complexity, maintaining mean reward above 0.94 at all complexity levels.

“CC Solver accuracy degrades gracefully with rule complexity, maintaining mean reward above 0.94 at all complexity levels.”

May combine two claims.

On SLR-Bench, QO Solver accuracy is 100.0% at ground-truth rule complexity 1 and 31.6% at complexity 4-5.

“| 1 | 50 | 100.0% | 1.0000 | 100.0% | 1.0000 |”

• Also stated in: § F.6 Additional Details: SLR-Bench · p. 32 · plain

  “| 4-5 | 250 | 44.4% | 0.9462 | 31.6% | 0.3160 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

BoK + CC and BoK + CC + QO do not differ in accuracy (∆ = 0.00, p = 1.000).

“| BoK + CC | BoK + CC + QO | +0.00 | 1.000 | +0.0000 | 1.000 |”

May combine two claims.

On PBEBench-Lite, the BoK vs. BoK+CC comparison shows a difference in accuracy of +0.10 pp and is not significant by either test (p = 1.0).

“BoK vs. BoK+CC ( ∆ Acc = +0 . 10 pp, p = 1 . 0 ) and BoK vs. BoK+All Sym ( ∆ Acc = +0 . 10 pp, p = 1 . 0 ) are not significant by either test, consistent with the ceiling effect.”

May combine two claims.

On SLR-Bench, CC Solver accuracy is 100.0% at ground-truth rule complexity 1 and 44.4% at complexity 4-5.

“| 1 | 50 | 100.0% | 1.0000 | 100.0% | 1.0000 |”

• Also stated in: § F.6 Additional Details: SLR-Bench · p. 32 · plain

  “| 4-5 | 250 | 44.4% | 0.9462 | 31.6% | 0.3160 |”

May combine two claims.

BoK+All Sym vs. BoK+CC+QO difference on PBEBench-Hard is +2.30 pp and significant (p<0.001).

“including the small BoK+All Sym vs. BoK+CC+QO difference ( +2 . 30 pp)”

May combine two claims.

On PBEBench-Lite, the BoK vs. DF comparison (∆ Acc = -1.49 pp) reaches conventional significance by McNemar (p = 0.020) but not by Wilcoxon (p = 0.595).

“The BoK vs. DF comparison ( ∆ Acc = -1 . 49 pp) reaches conventional significance by McNemar ( p = 0 . 020 ) but not by Wilcoxon ( p = 0 . 595 ), indicating that while BoK has a slight edge in binary solve rate, the two systems are essentially equivalent on partial-credit reward.”

May combine two claims.

On PBEBench-Lite, the BoK vs. BoK+All Sym comparison shows a difference in accuracy of +0.10 pp and is not significant by either test (p = 1.0).

“BoK vs. BoK+CC ( ∆ Acc = +0 . 10 pp, p = 1 . 0 ) and BoK vs. BoK+All Sym ( ∆ Acc = +0 . 10 pp, p = 1 . 0 ) are not significant by either test, consistent with the ceiling effect.”

Classified as probably not a substantial claim. The quote only partly states this claim.

The number of candidate rules (≤ 4-body) in the SLR-Bench search space grows from 30 at curriculum level 1 to 251,175 at curriculum level 20.

“| 20 | 50 | 251,175 | ∼ 3 . 5 hr | ∼ 14 hr |”

Probably not a substantial claim: may not pass S3 (load-bearing).

DF + CC and DF + CC + QO do not differ in accuracy (∆ = 0.10, p = 1.000).

“| DF + CC | DF + CC + QO | +0.10 | 1.000 | +0.0000 | 0.317 |”

Classified as probably not a substantial claim.

Each candidate requires a SWI-Prolog subprocess call taking 50-200 ms.

“Each candidate requires a SWI-Prolog subprocess call (50-200 ms)”

May combine two claims.

Reasoning traces can be compiled into reusable symbolic solvers that solve many tasks directly, complement LLM inference on hard cases, and provide a scalable route to domain-general solver induction.

“Together, these results show that reasoning traces can be compiled into reusable symbolic solvers that solve many tasks directly, complement LLMinference on hard cases, and provide a scalable route to domain-general solver induction.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The trace dataset of approximately 100 examples per benchmark is representative enough of the task distribution to induce generalizable solvers.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

Probably not a substantial claim: may not pass S3 (load-bearing).

For PBEBench, the complexity measure is computed only over correctly solved tasks because incorrect outputs are often degenerate and would bias the mean downward if included.

“For PBEBench, we follow the complexity measure defined in the original paper, but compute it only over correctly solved tasks : incorrect outputs are often degenerate and would bias the mean downward if included.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.

Probably not a substantial claim: may not pass S3 (load-bearing).

The agent operates over a sandboxed container containing the trace dataset and the verifier code with read-only access to avoid hacking the verifier.

“The agent operates over a sandboxed container containing the trace dataset and the verifier code (read-only access to avoid hacking the verifier).”

May combine two claims.

Hybrid inference substantially reduces LLM token usage, indicating that many tasks can be resolved through reusable symbolic structure instead of repeated LLM search.

“Hybrid inference substantially reduces LLM token usage, indicating that many tasks can be resolved through reusable symbolic structure instead of repeated LLM search.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

Classified as probably not a substantial claim. May combine two claims.

Evaluation is fully deterministic and verifier-based: a task is solved only when the reward is 1.0.

“Evaluation is fully deterministic and verifier-based: a task is solved only when the reward is 1.0.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.

May combine two claims.

On PBEBench-Lite, accuracy is already near saturation for LLM-based methods, leaving limited room for improvement.

“On PBEBench-Lite, accuracy is already near saturation for LLM-based methods, leaving limited room for improvement.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

The increased attempts and token usage on harder instances are often due to failures of compositional generalization and inefficient search.

“often due to failures of compositional generalization and inefficient search”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

Classified as probably not a substantial claim. May combine two claims.

The induction tasks are a small subset of the benchmark tasks; no ground-truth programs are accessed during induction.

“The induction tasks are a small subset of the benchmark tasks; no ground-truth programs are accessed during induction.”

May combine two claims.

When the solver fails, the LLM fallback is invoked independently and its partial-credit answers on hard tasks score lower than the solver's near-misses.

“when the solver fails, the LLM fallback is invoked independently and its partial-credit answers on hard tasks score lower than the solver's near-misses.”

May combine two claims.

Overall, these results indicate that even lightweight solvers suffice in saturated regimes, while more powerful solvers become critical on harder tasks.

“Overall, these results indicate that even lightweight solvers suffice in saturated regimes, while more powerful solvers become critical on harder tasks.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

May combine two claims.

Although the solvers recover some plausible sound laws, their tendency toward per-example patching is a formidable limitation for linguistic applications.

“Although the solvers do recover some plausible sound laws, this tendency is a formidable limitation for linguistic applications.”

Classified as probably not a substantial claim.

The coding agent observes only the input-output examples available to any solver at test time.

“The coding agent observes only the input-output examples available to any solver at test time.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

On PBEBench-Hard, we evaluate BoK only, as DF becomes prohibitively expensive and prone to lock-in for long cascades.

“On PBEBench-Hard, we evaluate BoK only, as DF becomes prohibitively expensive and prone to lock-in for long cascades.”

May combine two claims.

The token reductions are proportional to solver coverage and independent of per-token pricing, suggesting the efficiency gains scale to any deployment context.

“The token reductions are proportional to solver coverage and independent of per-token pricing, suggesting the efficiency gains scale to any deployment context.”

• AssumptionRests on an assumption: The verifier reward R correctly identifies correct programs.
• AssumptionRests on an assumption: Token usage is a valid proxy for computational cost and efficiency.
• AssumptionRests on an assumption: The constrained DSL is expressive enough to represent the ground-truth programs for the benchmarks.

Probably not a substantial claim: may not pass S4 (based).

It remains less explored whether the search behavior of an LLM itself can be compiled into a standalone solver that amortizes inference across tasks.

“What remains less explored is whether the search behavior of an LLM itself can be compiled into a standalone solver that amortizes inference across tasks.”

May combine two claims.

While accuracy is unaffected, over-complex programs are less interpretable and may generalize poorly to unseen distributions.

“While accuracy is unaffected, over-complex programs are less interpretable and may generalize poorly to unseen distributions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reported dollar costs are reference estimates computed using public API pricing for comparison purposes only; actual experiments were run on the cluster at no direct monetary cost.

“Reported dollar costs are reference estimates computed using public API pricing (DeepInfra for gpt-oss-120b ; AtlasCloud for Qwen3.6-35B-A3B ) for comparison purposes only ; actual experiments were run on the cluster at no direct monetary cost.”

May combine two claims.

12 examples with CoT is insufficient, suggesting a lower bound on the amount of demonstration diversity needed.

“Third, 12 examples with CoT is insufficient, suggesting a lower bound on the amount of demonstration diversity needed.”

May combine two claims.

Ensemble results over multiple runs are provided but depend on the number of induction attempts.

“ensemble results over multiple runs are provided but depend on the number of induction attempts.”

May combine two claims.

This variance reflects qualitatively different algorithms discovered by the coding agent across trajectories and is not reducible by fixing a random seed, as the agent's exploration is governed by its own sampling.

“This variance reflects qualitatively different algorithms discovered by the coding agent across trajectories and is not reducible by fixing a random seed, as the agent's exploration is governed by its own sampling.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Token counts are exact, taken from provider usage fields logged in every output JSONL.

“Token counts are exact, taken from provider usage fields logged in every output JSONL.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

As the frontier model's performance on these benchmarks approaches saturation, the accuracy benefit of symbolic solvers diminishes, and efficiency becomes the primary motivation.

“In particular, as the frontier model's performance on these benchmarks approaches saturation, the accuracy benefit of symbolic solvers diminishes, and efficiency becomes the primary motivation.”

May combine two claims.

The partial-credit gap between solvers and LLM scaling methods is smaller than the full-accuracy gap, confirming that solver failures are predominantly near-misses.

“The partial-credit gap between solvers and LLM scaling methods is smaller than the full-accuracy gap, confirming that solver failures are predominantly near-misses.”

Classified as probably not a substantial claim. May combine two claims.

Ensemble outputs are computed deterministically from individual outputs; no additional LLM calls are required.

“Ensemble outputs are computed deterministically from individual outputs; no additional LLM calls are required.”

May combine two claims.

Even the Lite search space is far beyond exhaustive enumeration: at 10^9 program evaluations per second and ≈10^7 seconds per year, the per-task-average Lite setting would require roughly 10^17 years; the Hard setting is cosmologically larger (≈10^190 years).

“Even the Lite search space is far beyond exhaustive enumeration: at 10 9 program evaluations per second and ≈ 10 7 seconds per year, the per-task-average Lite setting would require roughly 10 17 years; the Hard setting is cosmologically larger ( ≈ 10 190 years).”

May combine two claims.

The induced solvers are complementary to LLM search: they excel precisely where LLMs fail (long-horizon cascades, hard Prolog tiers) while LLMs cover the remaining tasks.

“The induced solvers are complementary to LLM search: they excel precisely where LLMs fail (long-horizon cascades, hard Prolog tiers) while LLMs cover the remaining tasks”

Classified as probably not a substantial claim. May combine two claims.

The CC Solver was induced in a single session per benchmark; no multi-run variance estimate is available for it.

“The CC Solver was induced in a single session per benchmark; no multi-run variance estimate is available for it.”

May combine two claims.

Performance reductions under the most extreme ablations, removing CoT entirely (Hard accuracy 24.8% vs. 51.8-74.7% for CoT runs), exceed the observed run-to-run algorithmic variance (up to 22.9pp on Hard), providing evidence that these ablation effects reflect some genuine differences rather than sampling noise alone.

“However, we note that the performance reductions under the most extreme ablations - removing CoT entirely (Hard accuracy 24.8% vs. 51.8-74.7% for CoT runs) - exceeds the observed run-to-run algorithmic variance (up to 22.9pp on Hard), providing evidence that these ablation effects reflect some genuine differences rather than sampling noise alone.”

May combine two claims.

Harder instances trigger more LLM attempts, compounding both cost and error.

“This shows that harder instances trigger more LLM attempts, compounding both cost and error.”

May combine two claims.

Weaker coding agents or smaller trace budgets may not successfully induce useful solvers, making the approach contingent on access to sufficiently capable code-generation models.

“Weaker coding agents or smaller trace budgets may not successfully induce useful solvers, making the approach contingent on access to sufficiently capable code-generation models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All BoK and DF baselines use a single model (gpt-oss-120b).

“All BoK and DF baselines use a single model (gpt-oss-120b).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Results may not generalize to other reasoning models, and the relative advantage of symbolic solvers may vary with model capability.

“Results may not generalize to other reasoning models, and the relative advantage of symbolic solvers may vary with model capability.”

May combine two claims. The quote only partly states this claim.

Three systematic causes account for most failures: (1) tasks where the correct program can only be discovered through a long intermediate chain (cascade length ≥18), where the beam horizon is exhausted; (2) all-four-BFCC tasks that require holding multiple mutually interacting ordering constraints simultaneously within the beam; (3) tasks where the pattern that fixes changed pairs also appears asa

“Three systematic causes account for most failures: (1) tasks where the correct program can only be discovered through a long intermediate chain (cascade length ≥ 18 ), where the beam horizon is exhausted before the necessary ordering is found; (2) all-four-BFCC tasks that require holding multiple mutually interacting ordering constraints simultaneously within the beam; (3) tasks where the pattern”

May combine two claims.

With 1-2 examples, multiple distinct cascades are consistent with the evidence, and any short program that happens to work scores 1.0 by coincidence.

“With 1-2 examples, multiple distinct cascades are consistent with the evidence, and any short program that happens to work scores 1.0 by coincidence.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The current training-accuracy reward is appropriate for PBEBench but misaligned for Real-FR.

“The current training-accuracy reward is appropriate for PBEBench (compact ground truth exists) but misaligned for Real-FR.”

• AssumptionRests on an assumption: PBEBench has compact ground truth.

May combine two claims.

BoK has a slight edge over DF in binary solve rate, but the two systems are essentially equivalent on partial-credit reward.

“indicating that while BoK has a slight edge in binary solve rate, the two systems are essentially equivalent on partial-credit reward.”

Classified as probably not a substantial claim.

The candidate count for SLR-Bench is (L 1) + (L 2) + (L 3) + (L 4), where L grows with curriculum level.

“The candidate count is ( L 1 ) + ( L 2 ) + ( L 3 ) + ( L 4 ) , where L grows with curriculum level.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The tendency of all systems on SLR-Bench to predict simpler rules than ground truth reflects the abundance of shorter equivalent Prolog forms.

“reflecting the abundance of shorter equivalent Prolog forms”

May combine two claims.

The non-significant BoK vs. BoK+CC and BoK vs. BoK+All Sym results on PBEBench-Lite are consistent with the ceiling effect.

“consistent with the ceiling effect.”

May combine two claims.

On PBEBench-Hard, the overall pass rates of BoK and CC Solver are similar while per-task rewards diverge sharply.

“so the overall pass rates are similar while per-task rewards diverge sharply”

May combine two claims.

DF+QO's accuracy gain comes from resolving tasks at the binary solve boundary without changing the reward distribution on partial-credit tasks.

“reflecting that DF+QO's accuracy gain comes from resolving tasks at the binary solve boundary without changing the reward distribution on partial-credit tasks”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

No multiple-comparison correction is applied to the statistical tests, and the tests are exploratory and intended to distinguish genuine performance gaps from sampling noise.

“No multiple-comparison correction is applied; these tests are exploratory and intended to distinguish genuine performance gaps from sampling noise.”