CoChat
Sign Up
PreprintCited by 25Software Engineering ResearchTeaching and Learning ProgrammingTopic Modeling

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning

Open access
Gehring, Jonas, Zheng, Kunhao, Copet, Jade, Mella, Vegard, Carbonneaux, Quentin, Cohen, Taco, Synnaeve, Gabriel·2024·arXiv
DOI 10.48550/arxiv.2410.02089arXiv 2410.02089
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new state-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps.

AI snapshot

Written from the paper’s full text.

TL;DR

RLEF is an end-to-end reinforcement learning method that trains code LLMs to use execution feedback over multiple turns, improving CodeContests solve rates while cutting required samples by an order of magnitude. It shows that grounding generations in automatic test feedback enables effective iterative code repair rather than relying on independent sampling.

Key findings

  • On CodeContests, RLEF improves Llama 3.1 Instruct 8B 1@3 solve rates from 8.9 to 17.2 on valid and 10.5 to 16.0 on test; Llama 3.1 Instruct 70B improves from 25.9 to 37.5 on valid and 27.5 to 40.1 on test.
  • At 10@100, RLEF improves 8B from 21.7 to 29.8 on valid and 24.8 to 28.7 on test; 70B improves from 50.2 to 54.5 on valid and 50.3 to 54.5 on test.
  • The 70B RLEF model beats AlphaCodium with GPT-4 on the test set with a single rollout compared to 5 solutions from 100 samples, and the 8B RLEF model is slightly ahead of AlphaCode 9B (16.0 vs 13.3) with a sample budget of 3 vs 1,000.
  • Analysis shows RLEF models make fewer initial errors, fix errors more reliably, and perform larger code edits; random execution feedback severely impairs repair, while gains also transfer to HumanEval+ and MBPP+.

Methods

  • Iterative code synthesis: the model repeatedly generates Python solutions from a problem description; public test execution feedback is appended after failures, and the final solution is evaluated on held-out private tests when public tests pass or a turn limit is reached.
  • RL setup: PPO optimizes the policy in a POMDP with binary reward +1 if all tests pass, -1 if any fail, and -0.2 for invalid code, plus a KL penalty (β = 0.05); no reward discounting (γ = 1), with token-level policy and turn-level value function.
  • Experiments: Llama 3.1 Instruct 8B and 70B trained on CodeContests (12,000 and 8,000 updates, turn limit 3), evaluated with n@k solve rates on 117 valid and 165 test problems, using 200 rollouts and temperatures 0.2 for 1@3 and 1.0 for 10@100.

Limitations

  • The paper states the code synthesis task is limited to improving a single solution to a given problem; generalizing to larger tasks requiring decomposition remains future work.
  • The paper states iterating on unit-test execution requires test cases, which may not be readily available; combining with automatic unit test generation is left as an avenue for further experiments.
  • Evident from the text: evaluation is centered on competitive programming (CodeContests) with additional transfer to HumanEval+ and MBPP+, and the method depends on public/private test splits and execution feedback formatting.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.02089

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 8 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Withholding public test execution feedback during training results in significantly worse performance.

    “In Appendix B.4 we show that withholding public test execution feedback during training results in significantly worse performance.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.
  • Findingown data

    Few-shot prompting and supervised fine-tuning from pre-trained Llama 3.1 models achieve significantly lower solve rates on CodeContests than the Instruct models in all cases.

    “We observe significantly lower performance compared to the Instruct models in all cases (Table 3a).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Maximizing test coverage for execution feedback at inference time can improve performance.

    “It may however be desirable to maximize test coverage for execution feedback at inference time, and we verify that this can indeed improve performance (Appendix B.2).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.
  • Findingown data

    For the 8B model prior to RLEF training, best performance can be obtained with independent samples (1 turn), with the exception of the test set above 30 samples.

    “For the 8B model (top row), prior to RLEF training, best performance can be obtained with independent samples (1 turn), with the exception of the test set above 30 samples.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Computing the probabilities of responses as the geometric mean rather than the product of token probabilities is beneficial for the KL penalty.

    “For the KL penalty, we found it beneficial to compute the probabilities of responses π(a t |c t ) as the geometric mean rather than product of token probabilities.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 43
  • Classified as probably not a substantial claim. 20
  • The quote only partly states this claim. 5

  • May combine two claims.

    Findingown data

    With random feedback, there is a clear impairment of self-repairs, demonstrating that RLEF allows LLMs to effectively leverage the provided feedback.

    “With random feedback, however, we see a clear impairment of self-repairs, demonstrating that RLEF allows LLMs to effectively leverage the provided feedback.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • May combine two claims.

    Findingown data

    On CodeContests, iterative code generation yields modest gains at best and drops in performance at worst, unless RLEF training is employed.

    “On CodeContests, iterative code generation yields modest gains at best and drops in performance at worst, unless RLEF training is employed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For the training duration of 12,000 gradient steps, the collected generations that do not pass public unit tests amount to 1.48M samples.

    “For the training duration of 12,000 gradient steps, this amounts to 1.48M samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On both valid and test sets, random feedback results in a drop in pass@1 which is amplified as the turn limit is increased.

    “On both valid and test sets, random feedback results in a drop in pass@1 which is amplified as the turn limit is increased.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • May combine two claims.

    Findingown data

    The model trained without public test execution feedback is better than the initial instruct model (12.2 vs 8.9 on validation and 10.9 vs 10.2 on test) but significantly below the RLEF-trained model (17.2 and 16.0).

    “This is better than the initial instruct model (8.9 and 10.2, respectively) but significantly below the corresponding RLEF-trained model (17.2 and 16.0, respectively).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Supervised fine-tuning improves Instruct model performance on the validation set only; no improvements are seen on the test set.

    “Supervised fine-tuning improves Instruct model performance on the validation set only; we do not see improvements on the test set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RLEF is also highly effective on the previously released 3.0 8B model, improving 1@3 solve rates from 4.1 to 12.5 and 3.2 to 12.1 on the valid and test set, respectively.

    “However, our RLEF method is also highly effective on the previously released 3.0 8B model, improving 1@3 solve rates from 4.1 to 12.5 and 3.2 to 12.1 on the valid and test set, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Fine-tuning models for next-token prediction with losses computed on the last response only produced slightly better models compared to training on all responses.

    “this produced slightly better models compared to training on all responses.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    RLEF training improves Llama 3.1 8B Instruct 1@3 solve rates on CodeContests from 8.9 to 17.2 on the valid set and from 10.5 to 16.0 on the test set.

    “| Llama 3.1 8B Instruct | Ours | 1@3 | 8.9 | 10.5 |”
    • Also stated in: § Result · plain
      “| + RLEF | Ours | 1@3 | 17.2 | 16.0 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    All problems in the validation and test set specify a memory limit, and only a few problems define a time limit.

    “All problems in the validation and test set specify a memory limit, and only a few problems define a time limit.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    RLEF training improves Llama 3.1 70B Instruct 10@100 solve rates on CodeContests from 50.2 to 54.5 on the valid set and from 50.3 to 54.5 on the test set.

    “| Llama 3.1 70B Instruct | Ours | 10@100 | 50.2 | 50.3 |”
    • Also stated in: § Result · plain
      “| + RLEF | Ours | 10@100 | 54.5 | 54.5 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the 70B RLEF model, including feedback from private and generated tests during inference improves validation set performance from 37.5 to 40.4 and test set performance from 38.0 to 41.2.

    “For the 70B RLEF model, validation set performance improves from 37.5 to 40.4, and on the test set we obtain 41.2 compared to 38.0 with feedback limited to public tests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After RLEF training, both the 8B and 70B models produce fewer wrong outputs in their initial response but are more prone to exceeding the allocated time limit.

    “We observe that after RLEF training, both the 8B (top row) and 70B (bottom row) models produce fewer wrong outputs in their initial response but are more prone to exceeding the allocated time limit.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • May combine two claims.

    Findingown data

    For pre-trained models, SFT yields improvements but lower scores compared to instruction-tuned models.

    “For pre-trained models, we see improvements from SFT but lower scores compared to instruction-tuned models (Appendix B.1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For each problem in the CodeContests training set, 100 multi-turn rollouts were collected, yielding 313,639 successful trajectories.

    “For each problem in the training set we collect 100 multi-turn rollouts and obtain 313,639 successful trajectories.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The 8B model with RLEF is slightly ahead compared to the similar-sized AlphaCode 9B model (16.0 and 13.3), but with a sample budget of 3 in our case and 1,000 for AlphaCode.

    “Likewise, the 8B model with RLEF is slightly ahead compared to the similar-sized AlphaCode 9B model (16.0 and 13.3), but with a sample budget of 3 in our case and 1,000 for AlphaCode.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The repair model combined with the single-turn RL model falls short of the RLEF-trained model that combines code synthesis and code repair (14.8 vs 17.2 on validation and 12.6 vs 16.0 on test).

    “but falls short of the corresponding RLEF-trained model which combines code synthesis and code repair (17.2 and 16.0, respectively; Table 1)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With random execution feedback, pass@1 is reduced while pass@10 suffers only slightly.

    “With random feedback pass@1 is reduced while pass@10 suffers only slightly”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    RLEF training improves Llama 3.1 70B Instruct 1@3 solve rates on CodeContests from 25.9 to 37.5 on the valid set and from 27.5 to 40.1 on the test set.

    “| Llama 3.1 70B Instruct | Ours | 1@3 | 25.9 | 27.5 |”
    • Also stated in: § Result · plain
      “| + RLEF | Ours | 1@3 | 37.5 | 40.1 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Together, the single-turn and repair model obtain 1@3 solve rates of 14.8 on the validation set and 12.6 on the test set; an improvement over the single-turn model alone (10.2 and 10.9) but significantly below the corresponding multi-turn model (17.2 and 16.0).

    “Together, the single-turn and repair model obtain 1@3 solve rates of 14.8 on the validation set and 12.6 on the test set; an improvement over the single-turn model alone (10.2 and 10.9) but significantly below the corresponding multi-turn model (17.2 and 16.0).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    RLEF improvements on CodeContests generalize to HumanEval+ and MBPP+, and to increased sample budgets compared to training time.

    “Our improvements from RLEF on CodeContests further generalize to HumanEval+ and MBPP+, two popular benchmarks for code synthesis, and to increased sample budgets compared to training time.”
    • Also stated in: § 5 CONCLUSION · plain
      “The RLEF-trained models further generalize to increased turn limits and to HumanEval+ and MBPP+, two popular code generation benchmarks that exhibit simpler programming questions and different execution feedback formatting.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The 70B model benefits from single-turn training and improves over multi-turn SFT results.

    “The 70B model benefits from single-turn training and improves over multi-turn SFT results in Table 3a.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    RLEF training improves Llama 3.1 8B Instruct 10@100 solve rates on CodeContests from 21.7 to 29.8 on the valid set and from 24.8 to 28.7 on the test set.

    “| Llama 3.1 8B Instruct | Ours | 10@100 | 21.7 | 24.8 |”
    • Also stated in: § Result · plain
      “| + RLEF | Ours | 10@100 | 29.8 | 28.7 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    RLEF achieves new state-of-the-art results on competitive programming tasks with both 8B and 70B models while reducing the number of samples required by an order of magnitude.

    “We benchmark on competitive programming tasks, where we achieve new state-of-the-art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reward function utilizes a held-out, private test set which is not accessible to the policy, so the MDP is partially observable.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Typically, a higher number of private tests and a large number of generated tests are available per problem in CodeContests.

    “typically, a higher number of private tests and a large number of generated tests are available per problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the 8B RLEF model, including feedback from private and generated tests during inference improves 1@3 solve rate on the valid set from 17.2 to 18.1 but drops on the test set from 16.0 to 14.4.

    “Comparing 1@3 solve rates (temperature 0.2) with a turn limit of 3, the 8B RLEF model can improve from 17.2 to 18.1 on the valid set, whereas on the test set we see a drop from 16.0 to 14.4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After RLEF, 3, 5 and 10 turns yield a consistent improvement over independent sampling, with best performance obtained with 5 turns.

    “After RLEF, we observe that 3, 5 and 10 turns yield a consistent improvement over independent sampling, with best performance obtained with 5 turns.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • May combine two claims.

    Findingown data

    Instruct models without RLEF perform only minimal code edits; they frequently output the same code solution despite inline feedback pointing out errors.

    “This underscores a shortcoming of the Instruct models without RLEF in that they perform only minimal code edits; indeed, we observe that they frequently output the same code solution despite inline feedback pointing out errors.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With our method, the self-repair capabilities of LLMs can be dramatically enhanced, resulting in superior performance of iterative code generation for both small and large sample budgets.

    “With our method, the self-repair capabilities of LLMs can be dramatically enhanced, resulting in superior performance of iterative code generation for both small and large sample budgets.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • May combine two claims.

    Findingown data

    RLEF is the most effective training method, followed by supervised fine-tuning (SFT).

    “RLEF is the most effective training method, followed by supervised fine-tuning (SFT).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the test set, the 70B model beats AlphaCodium with GPT-4, the previous state-of-the-art, with a single rollout compared to 5 solutions from 100 samples (38.0 and 29).

    “Notably, on the test set the 70B model beats AlphaCodium with GPT-4, the previous state-of-the-art, with a single rollout compared to 5 solutions from 100 samples (38.0 and 29).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The initial 70B model performs better with 3 or 5 turns, although, for small budgets, single turn performance is competitive.

    “The initial 70B model performs better with 3 or 5 turns, although, for small budgets, single turn performance is competitive.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With RLEF training, the models improve markedly on the original Llama 3.1 models and outperform prior works by a significant margin.

    “With RLEF training we improve markedly on the original Llama 3.1 models and outperform prior works by a significant margin.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reward function utilizes a held-out, private test set which is not accessible to the policy, so the MDP is partially observable.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The number of public test cases in the CodeContests validation and test sets varies between 1 and 7, with a median of 1.

    “The number of public test cases in the CodeContests validation and test sets vary between 1-7, with a median of 1”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Multi-turn gains from RLEF are most pronounced on CodeContests, the training domain of the models, but notable improvements are also observed on HumanEval+ and MBPP+.

    “While multi-turn gains from RLEF are most pronounced on CodeContests, the training domain of our models, we also observe notable improvements on HumanEval+ and MBPP+.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The Llama 3 family of models exhibit strong code generation performance out of the box and are able to follow instructions in the prompt, alleviating the need for an initial fine-tuning stage prior to RL training.

    “These models exhibit strong code generation performance out of the box and are able to follow instructions in the prompt, alleviating the need for an initial fine-tuning stage prior to RL training.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites AI @ Meta, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The code synthesis task considered is limited to improving a single solution to a given problem.

    “While our results demonstrate effective usage of inference-time feedback, the code synthesis task we consider is limited to improving a single solution to a given problem.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The transfer is attributed to the existent but comparabily weak multi-turn capabilities of the vanilla 70B Instruct model.

    “We attribute this to the existent but comparabily weak multi-turn capabilities of the vanilla 70B Instruct model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Main evaluations on CodeContests provide inference-time feedback on public test cases and estimate solve rates on private and generated tests.

    “Our main evaluations on CodeContests match our training setting, i.e., we provide inference-time feedback on public test cases and estimate solve rates on private (and the dataset's generated) tests.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    If specified, the memory and time limits are applied for RLEF training and evaluations; otherwise, a 1GB memory limit and maximum wall clock time of 10 seconds per test case are used.

    “If specified, we apply these limits for RLEF training and evaluations; otherwise, we use a 1GB memory limit and maximum wall clock time of 10 seconds per test case.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Models are evaluated after 200 optimizer steps with AdamW, and final parameters are selected based on validation set performance.

    “Models are evaluated after 200 optimizer steps with AdamW and we select final parameters based on validation set performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All models from the repair model sweep are evaluated in intervals of 400 gradient steps, and the best checkpoint is selected based on validation set performance.

    “We evaluate all models from the sweep in intervals of 400 gradient steps and select the best checkpoint based on validation set performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Kapoor et al. (2024) demonstrate that independent sampling beats the approaches from Shinn et al. (2023) and Zhong et al. (2024) when considering equal sampling budgets.

    “Kapoor et al. (2024) focus on inference cost and demonstrate that independent sampling beats the approaches from Shinn et al. (2023) and Zhong et al. (2024) when considering equal sampling budgets.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Kapoor et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Shinn et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zhong et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    With RLEF, samples within a rollout are of higher diversity (less similar codes) but edits are also targeted in that random execution feedback results in fewer successful repairs.

    “The analysis of Fig. 3 above suggests that, with RLEF, samples within a rollout are of higher diversity (less similar codes) but that edits are also targeted in that random execution feedback results in fewer successful repairs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Models are evaluated every 800 updates, and the final model is selected based on validation set performance.

    “Models are evaluated every 800 updates, and we select the final model based on validation set performance.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Intermediate feedback is crucial to ground LLM generations in the concrete situations encountered at inference time.

    “Intermediate feedback is therefore crucial to ground LLM generations in the concrete situations encountered at inference time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    This allows us to acquire both code synthesis and code repair capabilities with a single model, and to shift focus from large-sample inference regimes to obtaining high accuracy with low sample budgets.

    “This allows us to acquire both code synthesis and code repair capabilties with a single model, and to shift focus from large-sample inference regimes to obtaining high accuracy with low sample budgets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Le et al. (2022) implements program repair on top of an RL-trained LLM with two extra models: a critic that predicts the joint outcome of all unit tests and a repair model that maps wrong solutions to ground truth solutions.

    “Le et al. (2022) implements program repair on top of an RL-trained LLM with two extra models: a “critic” predicts the joint outcome of all unit tests (e.g., success, failure, runtime error) and can be used for ranking and determining promising prefixes, and a “repair” model maps wrong solutions to ground truth solutions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Le et al. (2022)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Our in-depth analysis revealed that, while an increase in correct first-turn generations and in the diversity of successive generations offers a major contribution of performance, our models also meaningfully take execution feedback into account and resolve errors over multiple turns.

    “Our in-depth analysis revealed that, while an increase in correct first-turn generations and in the diversity of successive generations offers a major contribution of performance, our models also meaningfully take execution feedback into account and resolve errors over multiple turns.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Public test execution feedback is sufficiently informative to improve performance on the private test set.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Olausson et al. (2024) show that sampling code solutions independently is competitive to repairing faulty code, that large models are required to provide effective feedback on errors, and that multiple rounds of repair are not effective.

    “Olausson et al. (2024) show that sampling code solutions independently is competitive to repairing faulty code, that large models are required to provide effective feedback on errors, and that multiple rounds of repair are not effective.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Olausson et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All experiments were performed with publicly available models and datasets.

    “We perform all experiments with publicly available models and datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Iterating on the execution results of unit tests naturally requires test cases, which may not be readily available.

    “Iterating on the execution results of unit tests naturally requires test cases, which may not be readily available.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The results provide further evidence that RL training of LLMs can reduce the diversity of outputs.

    “Kirk et al. (2024) observe that RL training of LLMs can reduce the diversity of outputs and we interpret our results as further evidence of their hypothesis.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kirk et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Candidate solutions are evaluated with the accompanying code-base of Li et al. (2022) using Python 3.10.

    “We evaluate candidate solutions with the accompanying code-base of Li et al. (2022) 3 using Python 3.10.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al. (2022)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

54 of 54

  • The authors state this as an assumption.

    The reward function utilizes a held-out, private test set which is not accessible to the policy, so the MDP is partially observable.

    “For generality, we assume a partially observable MDP as our reward function utilizes a held-out, private test set which is not accessible to the policy (unless an exact textual representation of the desired program behavior is provided in the problem description).”

  • The paper relies on this without support.

    Public test execution feedback is sufficiently informative to improve performance on the private test set.

    “Crucially, we utilize two different sets of test cases: a public test yields execution feedback that can be accessed during repeated attempts and forms the basis of selecting a final solution, whereas a private test set ultimately determines the correctness of the final solution.”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • 70B test solve rate differs between the results text and Table 1

    Major

    The main-results text reports a 38.0 test-set solve rate for the RLEF-trained 70B model, but Table 1 reports 40.1 for that model at 1@3.

  • Appendix B.4 misstates the initial 8B test solve rate

    Minor

    Appendix B.4 calls 10.2 the initial Instruct model's test-set solve rate. Table 1 reports 10.5 for the initial Llama 3.1 8B Instruct model.

  • Appendix comparison uses a different 70B public-feedback baseline

    Minor

    Appendix B.2 compares the 41.2 test-set result against 38.0 with public-test feedback, whereas Table 1 gives 40.1 for the corresponding RLEF-trained 70B result.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 15

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS, 2020.
  2. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching Large Language Models to Self-Debug. In ICLR, 2024b.
  3. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. In ICLR, 2020.
  4. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the Effects of RLHF on LLM Generalisation and Diversity. In ICLR, 2024.
  5. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In NeurIPS, 2022.
  6. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T. Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S. Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. StarCoder: May the source be with you! Transactions on Machine Learning Research, 2023.
  7. Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RLTF: Rein- forcement Learning from Unit Test Feedback. TMLR, 11/2023, 2023a.
  8. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
  9. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for General AI Assistants. In ICLR, 2024.
  10. Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Is Self-Repair a Silver Bullet for Code Generation? In ICLR, 2024.
  11. Maja Popović. chrF: Character n-Gram F-score for automatic MT evaluation. In WMT 2015, pp. 392-395, Lisbon, Portugal, 2015. Association for Computational Linguistics.
  12. Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. Execution-based Code Generation using Deep Reinforcement Learning. TMLR, 07/2023, 2023.
  13. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03924-6.
  14. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. In ICML, 2024.
  15. Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training Lan- guage Model Agents via Hierarchical Multi-Turn RL. In ICLR 2024 Workshop on Large Lan- guage Model (LLM) Agents, 2024.

Cited by

Sorted by citations

Related

Authors
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
267
Pages
19034-19055
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.02089
arXiv
2410.02089
OpenAlex
W4403882351
Semantic Scholar
585e95a43f4ceb3b9fdd8408b7b0b5df468c1030
Cited by
23
References
0
Citations by year
2026: 12 · 2025: 11
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Styrelsen för Internationellt Utvecklingssamarbete
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
execution feedback, reinforcement learning, program synthesis, competitive programming, LLM agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review