Teaching Large Language Models to Self-Debug
Open accessAbstract
Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.
AI snapshot
Written from the paper’s full text.
TL;DR
SELF-DEBUGGING teaches LLMs to debug their own generated code via few-shot prompting, using execution results and natural-language code explanations without additional training. It improves state-of-the-art accuracy on text-to-SQL, code translation, and text-to-Python benchmarks, and notably improves sample efficiency.
Key findings
- On Spider text-to-SQL, where no unit tests are available, SELF-DEBUGGING with code explanation improves the Codex baseline from 81.3 to 84.1, and improves accuracy on the hardest problems by 9%.
- On TransCoder C++-to-Python and MBPP text-to-Python, using unit tests plus code explanation improves baseline accuracy by up to 12%; e.g., Codex on TransCoder improves from 80.4 to 92.5 with UT + Expl.
- SELF-DEBUGGING improves sample efficiency: on Spider, applying it to greedy decoding matches the baseline using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline using 32 samples.
- Error analysis shows Spider fixes often address wrong WHERE conditions (25.7%), missing DISTINCT (17.1%), and wrong JOIN clauses (14.3%); on TransCoder and MBPP, 60%–70% of successful fixes address output mismatch errors.
Methods
- The method uses few-shot prompting with a pretrained LLM, without finetuning, in iterative turns of Generation, Explanation, and Feedback; feedback can be simple, unit-test-based, code-explanation-based, or execution-trace-based.
- For multiple initial samples, the paper selects the program with the most frequent execution result among those without execution errors, filtering by unit tests when available; maximum debugging turns is 10, though successful debugging mostly ends within 3 turns.
- Evaluation uses Codex, GPT-3.5, GPT-4, and StarCoder on Spider development set, TransCoder C++-to-Python with 560 problems and 10 unit tests each, and MBPP with 500 Python problems and 3 unit tests each, with the first MBPP unit test included in the prompt.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: on MBPP, context length limits reduce the number of exemplars when adding code explanation (6 exemplars for simple/unit-test feedback vs. 3 for unit test + explanation), and code explanation is less significant when the initial predicted code is far from correct.
- Evident from the text: GPT-4 is tuned for zero-shot prompting and does not follow few-shot exemplars as well as Codex on Spider; evaluation is limited to specific benchmarks and C++-to-Python translation rather than broader languages or tasks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
6 of 16 cited sources checked · 6 supported · 0 flagged
- Premisecitation
Large language models have been shown to be capable of describing their generated problem solutions in both text and code formats.
“On the other hand, large language models have been shown to be capable of describing their generated problem solutions in both text (Wei et al., 2022;Kojima et al., 2022;Zhou et al., 2023) and code (Gao et al., 2022;Chen et al., 2022) formats.”
Quote in the paperSubstantial claim1 of 2 sources checked - Findingown data
SELF-DEBUGGING consistently improves performance compared to prior code reranking approaches.
“We demonstrate that SELF-DEBUGGING consistently improves the performance.”
Quote in the paperSubstantial claim - Findingown data
It is insufficient for accurately predicting SQL correctness when the few-shot prompt only contains simple feedback.
“As will be shown in Section 5, it is insufficient for accurately predicting SQL correctness when the few-shot prompt only contains simple feedback.”
Quote in the paperNot a substantial claim - Findingown data
On Spider, applying SELF-DEBUGGING to predictions generated with greedy decoding matches the baseline accuracy using 16 samples.
“In particular, on Spider, applying SELF-DEBUGGING to predictions generated with greedy decoding matches the baseline accuracy using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline accuracy using 32 samples.”
Quote in the paperSubstantial claim - Findingown data
On the Spider benchmark, where no unit tests verify prediction correctness, SELF-DEBUGGING with code explanation consistently improves the baseline by 2-3%.
“On the Spider benchmark where there are no unit tests to verify the correctness of predictions, SELF-DEBUGGING with code explanation consistently improves the baseline by 2 -3%”
- Also stated in: § 1 INTRODUCTION · plain
“On the Spider benchmark (Yu et al., 2018) for text-to-SQL generation where there are no unit tests in the problem description, with code explanation, SELF-DEBUGGING consistently improves the baseline by 2 -3% with different numbers of initial programs”
Quote in the paperSubstantial claim - Also stated in: § 1 INTRODUCTION · plain
- Findingown data
Initial SQL queries tend to make small mistakes when questions require more complex SQL queries.
“In general, we observe that the initial SQL queries generated by LLMs are usually not completely wrong, but they tend to make small mistakes when the questions require more complex SQL queries, e.g., missing a few WHERE conditions or SELECT columns.”
Quote in the paperSubstantial claim - Findingown data
On extra hard problems, SELF-DEBUGGING increases accuracy by 9%.
“In particular, on extra hard problems, SELF-DEBUGGING increases the accuracy by 9%.”
Quote in the paperSubstantial claim- Rests on an assumption: The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
- Findingnone given
Without unit test execution, both models tend to be overconfident in their own initial predictions.
“Without unit test execution, both models tend to be overconfident in their own initial predictions, while GPT-4 is better than GPT-3.5 on Python generation.”
Quote in the paperSubstantial claim - Findingown data
Compared to Codex, few-shot prompting does not notably improve the SELF-DEBUGGING performance of GPT-3.5 and GPT-4.
“Compared to Codex, few-shot prompting does not notably improve the SELF-DEBUGGING performance of GPT-3.5 and GPT-4, and thus their SELF-DEBUGGING procedures fully rely on their internal code knowledge.”
Quote in the paperSubstantial claim - Findingown data
Leveraging code execution is crucial for enabling a performance leap with SELF-DEBUGGING.
“In Section 5.2.2, we further present an ablation study showing that leveraging code execution is crucial for enabling a performance leap with SELF-DEBUGGING.”
Quote in the paperSubstantial claim - Findingown data
Execution trace feedback consistently improves over simple feedback performance.
“With Codex, SELF-DEBUGGING still improves the performance by up to 5%, and the execution trace feedback consistently improves over the simple feedback performance.”
Quote in the paperSubstantial claim - Findingown data
SELF-DEBUGGING achieves state-of-the-art performance on several code generation benchmarks, including Spider for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation.
“SELF-DEBUGGING achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++to-Python translation, and MBPP for text-to-Python generation.”
Quote in the paperSubstantial claim - Findingown data
Typically one debugging turn is sufficient.
“Note that typically one debugging turn is sufficient, and the accuracy improvement after one turn is within 0.1%.”
Quote in the paperNot a substantial claim - Findingown data
Code explanation alone without debugging consistently improves code translation performance by 2-3%.
“and code explanation alone without debugging also consistently improves the code translation performance by 2 -3%”
Quote in the paperNot a substantial claim - Findingown data
Initial SQL queries generated by LLMs are usually not completely wrong.
“In general, we observe that the initial SQL queries generated by LLMs are usually not completely wrong, but they tend to make small mistakes when the questions require more complex SQL queries, e.g., missing a few WHERE conditions or SELECT columns.”
Quote in the paperSubstantial claim - Findingown data
GPT-4 is significantly better than Codex and GPT-3.5 for initial Python code generation on MBPP.
“GPT-4 is significantly better than Codex and GPT-3.5 for initial Python code generation on MBPP.”
Quote in the paperNot a substantial claim - Findingnone given
GPT-4 is better than GPT-3.5 on Python generation.
“Without unit test execution, both models tend to be overconfident in their own initial predictions, while GPT-4 is better than GPT-3.5 on Python generation.”
Quote in the paperNot a substantial claim - Findingown data
GPT-4 without unit test execution improves MBPP accuracy by 3.6%.
“GPT-4 without unit test execution improves the MBPP accuracy by 3.6%, and the improvement on other benchmarks is up to around 1%.”
Quote in the paperSubstantial claim - Findingreasoning
On the Spider benchmark, where unit tests are not available, simple feedback alone does not notably improve performance.
“On the Spider benchmark where unit tests are not available, simple feedback alone does not notably improve the performance”
Quote in the paperSubstantial claim - Findingown data
With Codex, SELF-DEBUGGING still improves performance by up to 5%.
“With Codex, SELF-DEBUGGING still improves the performance by up to 5%, and the execution trace feedback consistently improves over the simple feedback performance.”
Quote in the paperSubstantial claim - Findingown data
Sometimes LLMs can still improve their performance purely with self-generated feedback.
“In summary, code execution plays an important role in SELF-DEBUGGING; however, sometimes LLMs can still improve their performance purely with self-generated feedback.”
Quote in the paperNot a substantial claim - Findingown data
SELF-DEBUGGING fixes the prediction for an extra hard problem.
“Figure 7 presents an example where SELF-DEBUGGING fixes the prediction for an extra hard problem.”
Quote in the paperNot a substantial claim - Findingown data
GPT-4 without unit test execution improves other benchmarks by up to around 1%.
“GPT-4 without unit test execution improves the MBPP accuracy by 3.6%, and the improvement on other benchmarks is up to around 1%.”
Quote in the paperNot a substantial claim - Findingown data
Execution trace feedback offers consistent improvement.
“In particular, execution trace feedback offers consistent improvement.”
Quote in the paperSubstantial claim - Findingown data
Empirically, a large language model can benefit from rubber duck debugging, especially when unit tests are not available.
“Empirically, we observe that a large language model can also benefit from rubber duck debugging, especially when unit tests are not available.”
Quote in the paperSubstantial claim - Findingown data
The improvement achieved by SELF-DEBUGGING is more significant on hard problems.
“Figure 6b demonstrates that the improvement achieved by SELF-DEBUGGING is more significant on hard problems.”
Quote in the paperNot a substantial claim- Rests on an assumption: The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
- Findingown data
SELF-DEBUGGING notably improves sample efficiency when applied to different numbers of initial samples.
“Figure 6a demonstrates the effectiveness of SELF-DEBUGGING when applied to different numbers of initial samples, where SELF-DEBUGGING notably improves the sample efficiency.”
- Also stated in: § Self-debugging with explanation · plain
“SELF-DEBUGGING achieves the state-of-the-art performance across several code generation domains, and notably improves sample efficiency.”
Quote in the paperSubstantial claim - Also stated in: § Self-debugging with explanation · plain
- Findingown data
On Spider, SELF-DEBUGGING from 8 samples outperforms the baseline accuracy using 32 samples.
“In particular, on Spider, applying SELF-DEBUGGING to predictions generated with greedy decoding matches the baseline accuracy using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline accuracy using 32 samples.”
Quote in the paperSubstantial claim - Findingown data
On the Spider benchmark, SELF-DEBUGGING with code explanation improves prediction accuracy on the hardest-level problems by 9%.
“and improves the prediction accuracy on problems of the hardest level by 9%”
- Also stated in: § 1 INTRODUCTION · plain
“and improves the prediction accuracy on the most complicated SQL queries by 9%”
Quote in the paperSubstantial claim - Also stated in: § 1 INTRODUCTION · plain
- Findingown data
On TransCoder and MBPP, where unit tests are available, SELF-DEBUGGING improves baseline accuracy by up to 12%.
“On TransCoder and MBPP where unit tests are available, SELF-DEBUGGING improves the baseline accuracy by up to 12%.”
- Also stated in: § 1 INTRODUCTION · plain
“On both TransCoder for code translation (Roziere et al., 2020) and MBPP for text-to-Python generation (Austin et al., 2021), utilizing unit tests along with code explanation boosts the accuracy by up to 12%”
Quote in the paperSubstantial claim - Also stated in: § 1 INTRODUCTION · plain
- Findingown data
On TransCoder with gpt-3.5-turbo, 38.1% of errors fixed by SELF-DEBUGGING are runtime errors.
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
Quote in the paperNot a substantial claim - Findingown data
Incorporating both unit test execution and code explanation improves debugging performance.
“Meanwhile, incorporating both unit test execution and code explanation improves the debugging performance, and we present some examples in Figures 9 and 10 in Appendix D.”
Quote in the paperNot a substantial claim - Findingown data
Leveraging code explanation alone without SELF-DEBUGGING provides a consistent performance gain of 2-3% for different numbers of samples.
“In addition, we demonstrate that leveraging code explanation alone without SELF-DEBUGGING also provides a consistent performance gain of 2 -3% for different numbers of samples, as shown in Figure 8b.”
Quote in the paperNot a substantial claim - Findingown data
When using greedy decoding, GPT-3.5 achieves an accuracy of 71.1% with few-shot prompting on Spider.
“For example, when using greedy decoding, GPT-3.5 achieves an accuracy of 59.9% with zero-shot prompting, and 71.1% with few-shot prompting, which is over 6% lower than Codex.”
Quote in the paperSubstantial claim - Findingown data
With greedy decoding, SELF-DEBUGGING increases accuracy by over 8% on MBPP.
“With greedy decoding, SELF-DEBUGGING increases the accuracy by over 8%, and leveraging both unit test execution and code explanation again consistently improves the performance.”
Quote in the paperSubstantial claim - Findingown data
On the MBPP dataset, SELF-DEBUGGING with unit test and explanation feedback achieves 75.6 accuracy.
“UT UT + Expl. | 75.2 75.6”
Quote in the paperSubstantial claim - Findingown data
On the MBPP dataset, SELF-DEBUGGING with Simple feedback achieves 73.6 accuracy.
“Simple | 73.6”
Quote in the paperSubstantial claim - Findingown data
On MBPP, 2/3 of the initial programs have problem-specific semantic errors.
“On MBPP, while 2/3 of the initial programs have problem-specific semantic errors, over 10% of the initial programs can be fixed by switching the order of function arguments and matching the argument types.”
Quote in the paperNot a substantial claim - Findingown data
On the TransCoder dataset, TransCoder achieves 44.5 pass@1.
“TransCoder | 44.5”
Quote in the paperNot a substantial claim - Findingown data
On the MBPP dataset, Reviewer achieves 66.9 accuracy with 25 samples.
“Reviewer | 66.9 (n = 25)”
Quote in the paperNot a substantial claim - Findingown data
On MBPP, over 10% of the initial programs can be fixed by switching the order of function arguments and matching the argument types.
“On MBPP, while 2/3 of the initial programs have problem-specific semantic errors, over 10% of the initial programs can be fixed by switching the order of function arguments and matching the argument types.”
Quote in the paperNot a substantial claim - Findingown data
The major improvement comes from the first debugging turn.
“From Figure 8a, we again observe that the major improvement comes from the first debugging turn.”
Quote in the paperNot a substantial claim - Findingown data
SELF-DEBUGGING performance of Codex is on par with GPT-3.5 on Transcoder.
“From Tables 5b and 5c, we show that while GPT-3.5 notably outperforms Codex on generating initial code in Python, the SELF-DEBUGGING performance of Codex is on par with GPT-3.5 on Transcoder.”
Quote in the paperNot a substantial claim - Findingown data
SELF-DEBUGGING achieves state-of-the-art performance across several code generation domains.
“SELF-DEBUGGING achieves the state-of-the-art performance across several code generation domains, and notably improves sample efficiency.”
Quote in the paperSubstantial claim - Findingown data
On the Spider development set with training, T5-3B + N-best Reranking achieves 80.6 accuracy.
“T5-3B + N-best Reranking | 80.6”
Quote in the paperNot a substantial claim - Findingown data
Codex outperforms GPT-3.5 on SELF-DEBUGGING for text-to-SQL generation.
“In Table 5a, we observe that Codex also outperforms GPT-3.5 on SELF-DEBUGGING for text-to-SQL generation, and code explanation again improves the performance for GPT-3.5.”
Quote in the paperNot a substantial claim - Findingown data
On the TransCoder dataset, SELF-DEBUGGING with Codex achieves 80.4 pass@1.
“Codex | 80.4”
Quote in the paperSubstantial claim - Findingown data
Code explanation plays a smaller role when the initial predicted code is far from the correct solution.
“Third, we find that code explanation plays a smaller role when the initial predicted code is far from the correct solution, in which case the model simply generates a new program and does not rely on code snippets in the initial prediction.”
Quote in the paperNot a substantial claim - Findingown data
GPT-3.5 SELF-DEBUGGING performance with few-shot exemplars is similar to zero-shot prompting.
“We present GPT-3.5 SELF-DEBUGGING results using zero-shot prompting, as we observe that the performance with few-shot exemplars is similar.”
Quote in the paperSubstantial claim - Findingown data
When using greedy decoding, GPT-3.5 achieves an accuracy of 59.9% with zero-shot prompting on Spider.
“For example, when using greedy decoding, GPT-3.5 achieves an accuracy of 59.9% with zero-shot prompting, and 71.1% with few-shot prompting, which is over 6% lower than Codex.”
Quote in the paperSubstantial claim - Findingown data
For code translation and text-to-Python generation tasks where unit tests are available, SELF-DEBUGGING significantly increases baseline accuracy by up to 12%.
“For code translation and text-to-Python generation tasks where unit tests are available, SELF-DEBUGGING significantly increases the baseline accuracy by up to 12%.”
Quote in the paperSubstantial claim - Findingown data
Applying SELF-DEBUGGING to greedy decoding is close to baseline accuracy with 10 samples.
“Compared to Figure 8b, applying SELF-DEBUGGING to greedy decoding outperforms the baseline accuracy with 5 samples, and is close to the baseline accuracy with 10 samples.”
Quote in the paperNot a substantial claim - Findingown data
On the MBPP dataset, SELF-DEBUGGING with unit test feedback achieves 75.2 accuracy.
“UT UT + Expl. | 75.2 75.6”
Quote in the paperSubstantial claim - Findingown data
On the MBPP dataset, LEVER achieves 68.9 accuracy with 100 samples.
“LEVER | 68.9 (n = 100)”
Quote in the paperNot a substantial claim - Findingown data
On Spider, the performance of GPT-3.5 is significantly worse than Codex.
“On Spider, we observe that the performance of GPT-3.5 is significantly worse than Codex.”
Quote in the paperNot a substantial claim - Findingown data
Applying SELF-DEBUGGING to greedy decoding outperforms baseline accuracy with 5 samples.
“Compared to Figure 8b, applying SELF-DEBUGGING to greedy decoding outperforms the baseline accuracy with 5 samples, and is close to the baseline accuracy with 10 samples.”
Quote in the paperNot a substantial claim - Findingown data
SELF-DEBUGGING fixes the prediction for an extra hard problem on the Spider benchmark.
“Figure 7: An example on the Spider benchmark, where SELF-DEBUGGING fixes the prediction for an extra hard problem.”
Quote in the paperNot a substantial claim - Findingcitation
On the Spider development set with training, LEVER achieves 81.9 accuracy.
“LEVER (Ni et al., 2023) | 81.9”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ni et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Codex performs close to GPT-3.5 on MBPP.
“It also performs close to GPT-3.5 on MBPP, and outperforms it for both Simple and UT.”
Quote in the paperNot a substantial claim - Findingown data
On Transcoder, over 30% of successful fixes address implementation differences between different programming languages.
“Specifically, on Transcoder, over 30% of successful fixes address implementation differences between different programming languages, where the issue is made more obvious with code execution.”
Quote in the paperNot a substantial claim - Findingown data
On the Spider development set, MBR-Exec achieves 75.2 accuracy.
“MBR-Exec | 75.2”
Quote in the paperNot a substantial claim - Findingown data
On the TransCoder dataset, PaLM achieves 51.8 pass@1.
“PaLM | 51.8”
Quote in the paperNot a substantial claim - Findingown data
On Transcoder and MBPP, 60%-70% of successful cases fix the output mismatch error when the initial wrong code is very close to a correct one.
“On the other hand, on Transcoder and MBPP, 60%-70% successful cases fix the output mismatch error when the initial wrong code is very close to a correct one, as shown in Table 4b.”
Quote in the paperNot a substantial claim - Findingown data
On the MBPP dataset, MBR-Exec achieves 63.0 accuracy with 25 samples.
“MBR-Exec | 63.0 (n = 25)”
Quote in the paperNot a substantial claim - Findingown data
On the TransCoder dataset, PaLM-Coder achieves 55.1 pass@1.
“PaLM-Coder | 55.1”
Quote in the paperNot a substantial claim - Findingown data
Leveraging both unit test execution and code explanation consistently improves performance on MBPP.
“With greedy decoding, SELF-DEBUGGING increases the accuracy by over 8%, and leveraging both unit test execution and code explanation again consistently improves the performance.”
Quote in the paperSubstantial claim - Findingown data
On text-to-SQL generation, leveraging code explanation for SELF-DEBUGGING provides a performance gain of 9% on the hardest problems.
“On text-to-SQL generation where there are no unit tests specified for the task, leveraging code explanation for SELF-DEBUGGING consistently improves the baseline by 2 -3%, and provides a performance gain of 9% on the hardest problems.”
Quote in the paperSubstantial claim - Findingown data
On MBPP with gpt-4, 69.2% of errors fixed by SELF-DEBUGGING are output mismatches.
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
Quote in the paperNot a substantial claim - Findingown data
On the TransCoder dataset, SELF-DEBUGGING with unit test and explanation feedback achieves 92.5 pass@1.
“UT + Expl. | 92.5”
Quote in the paperSubstantial claim - Findingown data
On the MBPP dataset, SELF-DEBUGGING with Codex achieves 72.2 accuracy with 10 samples.
“Codex | 72.2 (n = 10)”
Quote in the paperNot a substantial claim - Findingown data
GPT-3.5 notably outperforms Codex on generating initial code in Python.
“From Tables 5b and 5c, we show that while GPT-3.5 notably outperforms Codex on generating initial code in Python, the SELF-DEBUGGING performance of Codex is on par with GPT-3.5 on Transcoder.”
Quote in the paperNot a substantial claim - Findingown data
Preliminary results suggest that model-generated feedback messages about semantic errors do not provide additional benefits on top of line-by-line code explanation.
“Our preliminary results suggest that modelgenerated feedback messages about semantic errors do not provide additional benefits on top of line-by-line code explanation, and future work can explore techniques to predict more informative error messages.”
Quote in the paperNot a substantial claim - Findingown data
Code explanation improves the performance for GPT-3.5.
“In Table 5a, we observe that Codex also outperforms GPT-3.5 on SELF-DEBUGGING for text-to-SQL generation, and code explanation again improves the performance for GPT-3.5.”
Quote in the paperNot a substantial claim - Findingown data
A single debugging turn with full feedback improves over greedy decoding accuracy by around 12%.
“Specifically, a single debugging turn with the full feedback improves over the greedy decoding accuracy by around 12%.”
Quote in the paperSubstantial claim - Findingreasoning
The improvement using code explanation is less significant on MBPP.
“We observe that the improvement using code explanation is less significant on MBPP.”
Quote in the paperNot a substantial claim - Findingown data
On text-to-SQL generation where there are no unit tests specified for the task, leveraging code explanation for SELF-DEBUGGING consistently improves the baseline by 2-3%.
“On text-to-SQL generation where there are no unit tests specified for the task, leveraging code explanation for SELF-DEBUGGING consistently improves the baseline by 2 -3%, and provides a performance gain of 9% on the hardest problems.”
Quote in the paperSubstantial claim - Findingown data
On the Spider development set, Coder-Reviewer achieves 74.5 accuracy.
“Coder-Reviewer | 74.5”
Quote in the paperNot a substantial claim - Findingown data
GPT-3.5's performance on Spider is over 6% lower than Codex.
“For example, when using greedy decoding, GPT-3.5 achieves an accuracy of 59.9% with zero-shot prompting, and 71.1% with few-shot prompting, which is over 6% lower than Codex.”
Quote in the paperNot a substantial claim - Premisecitation
Prior work has yet to show the effectiveness of model-generated feedback on code generation.
“prior work has yet to show the effectiveness of model-generated feedback on code generation (Chen et al., 2023a).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationnone given
SELF-DEBUGGING teaches a large language model to debug its predicted program via few-shot demonstrations.
“In this work, we propose SELF-DEBUGGING, which teaches a large language model to debug its predicted program via few-shot demonstrations.”
Quote in the paperSubstantial claim - Interpretationown data
Code execution plays an important role in SELF-DEBUGGING.
“In summary, code execution plays an important role in SELF-DEBUGGING; however, sometimes LLMs can still improve their performance purely with self-generated feedback.”
Quote in the paperSubstantial claim - Interpretationreasoning
SELF-DEBUGGING with code explanation facilitates the LLM to identify the discrepancy between the question and the predicted SQL query, resulting in an accuracy boost for more complex tasks.
“In this case, SELF-DEBUGGING with code explanation facilitates the LLM to identify the discrepancy between the question and the predicted SQL query, resulting in an accuracy boost for more complex tasks.”
Quote in the paperSubstantial claim - Interpretationreasoning
GPT-3.5 and GPT-4 SELF-DEBUGGING procedures fully rely on their internal code knowledge.
“Compared to Codex, few-shot prompting does not notably improve the SELF-DEBUGGING performance of GPT-3.5 and GPT-4, and thus their SELF-DEBUGGING procedures fully rely on their internal code knowledge.”
Quote in the paperSubstantial claim - Interpretationnone given
Due to availability of unit tests, it is much easier to recognize wrong predictions from unit test feedback.
“First, due to the availability of unit tests, it is much easier to recognize wrong predictions from unit test feedback.”
Quote in the paperSubstantial claim - Interpretationown data
SELF-DEBUGGING enables a large language model to debug code generated by itself.
“In this work, we presented SELF-DEBUGGING, which enables a large language model to debug code generated by itself.”
Quote in the paperSubstantial claim - Interpretationnone given
SELF-DEBUGGING improves sample efficiency compared to utilizing execution solely for initial code generation.
“In this work, our main focus is to utilize and explain code execution for SELF-DEBUGGING, which improves the sample efficiency compared to utilizing execution solely for initial code generation.”
Quote in the paperSubstantial claim - Interpretationown data
SELF-DEBUGGING prompt with few-shot exemplars is effective.
“These results demonstrate the effectiveness of our SELF-DEBUGGING prompt with few-shot exemplars, while suggesting that Codex is better than GPT-3.5 at learning from few-shot exemplars with long context.”
Quote in the paperSubstantial claim - Interpretationown data
On Transcoder, the issue of implementation differences is made more obvious with code execution.
“Specifically, on Transcoder, over 30% of successful fixes address implementation differences between different programming languages, where the issue is made more obvious with code execution.”
Quote in the paperNot a substantial claim - Interpretationown data
When initial predicted code is far from correct solution, the model simply generates a new program and does not rely on code snippets in the initial prediction.
“Third, we find that code explanation plays a smaller role when the initial predicted code is far from the correct solution, in which case the model simply generates a new program and does not rely on code snippets in the initial prediction.”
Quote in the paperNot a substantial claim - Interpretationown data
Codex is better than GPT-3.5 at learning from few-shot exemplars with long context.
“These results demonstrate the effectiveness of our SELF-DEBUGGING prompt with few-shot exemplars, while suggesting that Codex is better than GPT-3.5 at learning from few-shot exemplars with long context.”
Quote in the paperNot a substantial claim
- May combine two claims. 18
- Classified as probably not a substantial claim. 27
- The quote only partly states this claim. 9
May combine two claims.
InterpretationcitationDifferent from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them, SELF-DEBUGGING teaches the model to identify implementation errors by investigating execution results and explaining the code by itself.
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Austin et al., 2021 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationDecoding multiple samples can significantly improve the performance of large language models.
“Prior works demonstrate that decoding multiple samples can significantly improve the performance of large language models (Wang et al., 2023;Shi et al., 2022).”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Wang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Shi et al., 2022 source
Classified as probably not a substantial claim.
PremisecitationFor code generation tasks, code execution can be utilized to select the final prediction.
“In particular, for code generation tasks, we can utilize code execution to select the final prediction (Chen et al., 2019;Li et al., 2022;Shi et al., 2022;Zhang et al., 2022;Ni et al., 2023).”
Quote in the paperSubstantial claim1 of 3 sources checked- cites Chen et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Li et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Shi et al., 2022 source
Classified as probably not a substantial claim.
PremisecitationPrior work on code repair has demonstrated that training the repair model on execution traces improves debugging performance.
“Prior work on code repair has demonstrated that training the repair model on execution traces improves the debugging performance (Wang et al., 2018;Gupta et al., 2020).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Wang et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Gupta et al., 2020 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationFew-shot prompting instructs the language model to solve a task with several input-output demonstrations.
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Brown et al., 2020 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenSimple feedback does not produce meaningful changes to the initial predicted SQL queries.
“and does not produce meaningful changes to the initial predicted SQL queries.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingreasoningOn TransCoder and MBPP, where at least one unit test is available, simple feedback also utilizes the execution result to infer code correctness even if execution information is not presented in the feedback message, so simple feedback still improves model prediction for both applications.
“Note that on TransCoder and MBPP benchmarks where at least one unit test is available for SELF-DEBUGGING, simple feedback also utilizes the execution result to infer the code correctness even if the execution information is not presented in the feedback message, thus simple feedback still improves the model prediction for both applications.”
Quote in the paperSubstantial claimMay combine two claims.
FindingreasoningOn Spider, both GPT-4's initial SQL generation and SELF-DEBUGGING performance are much worse than Codex.
“Although GPT-4 is shown to be much stronger than previous GPT models on many tasks (Ope-nAI, 2023), on Spider, both its initial SQL generation and SELF-DEBUGGING performance are much worse than Codex.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataDespite StarCoder's baseline performance being considerably worse than GPT models, SELF-DEBUGGING with unit test execution offers a significant performance gain, i.e., 6% on MBPP with execution trace feedback.
“Despite that the baseline performance of StarCoder is considerably worse than GPT models, SELF-DEBUGGING with unit test execution also offers a significant performance gain, i.e., 6% on MBPP with the execution trace feedback.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor all tasks, models generally benefit from richer feedback for SELF-DEBUGGING, especially when execution information is present in the feedback.
“For all tasks, models generally benefit from richer feedback for SELF-DEBUGGING, especially when the execution information is present in the feedback.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataBy leveraging feedback messages and reusing failed predictions, SELF-DEBUGGING notably improves sample efficiency and can match or outperform baseline models that generate more than 10× candidate programs.
“Meanwhile, by leveraging feedback messages and reusing failed predictions, SELF-DEBUGGING notably improves sample efficiency, and can match or outperform baseline models that generate more than 10× candidate programs.”
- Also stated in: § 1 INTRODUCTION · plain
“Meanwhile, SELF-DEBUGGING improves sample efficiency, and can match or outperform baseline models that sample more than 10× predictions.”
Quote in the paperSubstantial claim- Also stated in: § 1 INTRODUCTION · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataSuccessful debugging processes mostly end within 3 turns.
“though empirically the successful debugging processes mostly end within 3 turns”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenSimilar sample efficiency improvement is observed on other benchmarks.
“We observe similar sample efficiency improvement on other benchmarks, and we defer the discussion to Appendix C.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenThe accuracy improvement after one debugging turn is within 0.1%.
“Note that typically one debugging turn is sufficient, and the accuracy improvement after one turn is within 0.1%.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingown dataLeveraging unit tests whenever available consistently boosts SELF-DEBUGGING performance.
“In our experiments, we will demonstrate that leveraging unit tests whenever available consistently boosts the SELF-DEBUGGING performance.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataGPT-4's self-debugging performance gain is on par with Codex: over 12% improvement on TransCoder and 8% improvement on MBPP.
“Meanwhile, its self-debugging performance gain is on par with Codex: over 12% improvement on Transcoder, and 8% improvement on MBPP.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataSELF-DEBUGGING can teach the large language model to perform rubber duck debugging; without any human feedback on code correctness or error messages, the model is able to identify its mistakes by investigating execution results and explaining the generated code in natural language.
“In particular, we demonstrate that SELF-DEBUGGING can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 5.7% of errors due to wrong COUNT columns.
“Wrong COUNT columns | 5.7”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataSELF-DEBUGGING with unit test feedback fixes the code translation error, while simple feedback fails.
“Figure 9: An example where SELF-DEBUGGING with unit test feedback fixes the code translation error, while the simple feedback fails.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe translation with code explanation correctly checks for the end of a string.
“Specifically, the translation with code explanation correctly checks for the end of a string.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 14.3% of errors due to wrong JOIN clauses.
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 5.7% of errors due to missing nested conditions.
“Missing nested conditions | 5.7”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataOn TransCoder with gpt-3.5-turbo, 61.9% of errors fixed by SELF-DEBUGGING are output mismatches.
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 25.7% of errors due to wrong WHERE conditions.
“Wrong WHERE conditions | 25.7”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 8.6% of errors due to wrong INTERSECT/UNION clauses.
“Wrong INTERSECT/UNION clauses | 8.6”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataOn MBPP with gpt-4, 30.8% of errors fixed by SELF-DEBUGGING are runtime errors.
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 5.7% of errors due to wrong column selection.
“Wrong column selection | 5.7”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataSELF-DEBUGGING makes major changes to the initial code in Figure 11 example.
“See Figure 11 in Appendix D for an example where SELF-DEBUGGING makes major changes to the initial code.”
- Also stated in: § Original Python code · plain
“Figure 11: An example on MBPP where the prediction after SELF-DEBUGGING is very different from the initial code.”
Quote in the paperNot a substantial claim- Also stated in: § Original Python code · plain
Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 11.4% of errors due to wrong number of SELECT columns.
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAdding code explanation results in correct code translation, while baseline prompt does not.
“Figure 10: An example where adding code explanation results in the correct code translation, while the baseline prompt does not.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 5.8% of errors due to wrong aggregate functions and keywords.
“Wrong aggregate functions and keywords 5.8”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataCodex outperforms GPT-3.5 for both Simple and UT on MBPP.
“It also performs close to GPT-3.5 on MBPP, and outperforms it for both Simple and UT.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataOn the Spider development set, SELF-DEBUGGING fixes 17.1% of errors due to missing the DISTINCT keyword.
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationPrior works suggest that large language models are not yet capable of correcting code when lacking external feedback, such as unit tests or human instructions.
“prior works suggest that such large language models are not yet capable of correcting code when lacking external feedback, such as unit tests or human instructions (Chen et al., 2023a).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningIn text-to-SQL generation, SELF-DEBUGGING enables the model to identify errors and justify code correctness by explaining the predicted code.
“First, text-to-SQL generation represents the scenario where no unit tests are available, where SELF-DEBUGGING enables the model to identify errors and justify code correctness by explaining the predicted code.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationRubber duck debugging for human programmers, where explaining code line-by-line in natural language to a rubber duck significantly boosts debugging efficiency without expert guidance.
“This debugging process is reminiscent of rubber duck debugging for human programmers, where explaining the code line-byline in natural language to a rubber duck significantly boosts debugging efficiency without expert guidance (Hunt & Thomas, 2000).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hunt & Thomas, 2000
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Interpretationnone givenSince unit tests are not available, it becomes more challenging for the model to infer the correctness of predicted SQL queries.
“Since the unit tests are not available, it becomes more challenging for the model to infer the correctness of the predicted SQL queries.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningBoth the execution trace and the line-by-line explanation come from model generation instead of code execution, so the trace feedback does not require more information than the pure code explanation feedback and does not require access to intermediate execution states.
“Note that both the execution trace and the line-by-line explanation come from model generation instead of code execution, thus the trace feedback does not require more information than the pure code explanation feedback; i.e., no access to intermediate execution states.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationnone givenSELF-DEBUGGING empowers the model to perform rubber duck debugging, so that the model can identify and fix bugs without human instructions.
“In particular, we demonstrate that SELF-DEBUGGING empowers the model to perform rubber duck debugging, so that the model can identify and fix bugs without human instructions.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenAfter the success feedback round, SELF-DEBUGGING is terminated and no further 'Feedback for failure' messages are issued, even if the new prediction is incorrect.
“(Note that after this kind of feedback, we finish SELF-DEBUGGING and do not continue with”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone givenSELF-DEBUGGING teaches the large language model to notice prediction errors via code explanation in the style of rubber duck debugging, which does not require the feedback message to explicitly explain the implementation errors and how to fix them.
“In particular, SELF-DEBUGGING teaches the large language model to notice prediction errors via code explanation in the style of rubber duck debugging, which does not require the feedback message to explicitly explain the implementation errors and how to fix them.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenIn the chat feedback prompts, a new code prediction produced after the success feedback round is used for evaluation only if it also passes the unit test; otherwise the original prediction that passed is used.
“The new code prediction is used only if it also passes the unit test, otherwise the original prediction which passes will be used for evaluation.”
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
“The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.”
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
“The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.”
Quote in the paperSubstantial claim- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
Classified as probably not a substantial claim.
Interpretationnone givenOur work highlights the promise of improving the coding performance of large language models by teaching them to iteratively debug their own predictions, instead of requiring the model to generate the correct code from scratch.
“Our work highlights the promise of improving the coding performance of large language models by teaching them to iteratively debug their own predictions, instead of requiring the model to generate the correct code from scratch.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataSELF-DEBUGGING demonstrates the effectiveness of both self-generated feedback and unit test feedback acquired by code execution.
“In this work, we focus on code generation applications, and SELF-DEBUGGING demonstrates the effectiveness of both self-generated feedback and unit test feedback acquired by code execution.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationnone givenSELF-DEBUGGING enables the model to generate feedback messages on its own at test time, and does not require extra training.
“On the other hand, SELF-DEBUGGING enables the model to generate feedback messages on its own at test time, and does not require extra training.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
For complex programming tasks, generating the correct solution in one go is challenging.
“However, for complex programming tasks, generating the correct solution in one go becomes challenging”
“However, for many programming tasks, generating correct code with a single attempt is challenging.”
The paper relies on this without support.
Human programmers typically look into incorrect code and investigate execution results, then make changes to resolve implementation errors.
“Instead of completely discarding the incorrect code, humans typically look into the code and investigate execution results, then make changes to resolve the implementation errors.”
The paper relies on this without support.
There is no guarantee that code written on the first try by human programmers is always accurate.
“Intuitively, even for human programmers, there is no guarantee that the code written on the first try is always accurate.”
The paper relies on this without support.
Human programmers can usually identify bugs by describing code implementation and comparing it to the problem description without extra guidance.
“By describing the code implementation and comparing it to the problem description, human programmers are usually able to identify the bugs without extra guidance.”
The paper relies on this without support.
The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
“where the difficulty of each problem is annotated in the Spider benchmark based on the complexity of the ground truth SQL queries.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AlphaCode, Self-Debugging and CodeRL rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [7]
The paper contradicts both assertions about Self-Debugging. Its framework section says feedback can come from the model itself rather than unit tests, and §5.2.2 reports improvement without unit-test execution. The “Self-debugging with explanation” section says the model identifies errors through its own code explanations and uses feedback to revise its code. The cited paper does not establish the broader assertions about AlphaCode, CodeRL, or pass@1 eligibility. (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredSelf-Check with 3-shot CoT and zero-shot CoT have shown to been effective on various tasks including dialogue response, math reasoning, and code generation.
“We evaluate FACTOOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023;Chen et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Chen et al., 2023
Section C.1 (page unavailable) reports that self-debugging improves code-translation accuracy, supporting the code-generation portion of the claim. The paper does not establish that the named Self-Check 3-shot CoT and zero-shot CoT baselines are effective across dialogue response, math reasoning, and code generation; its discussion of reasoning tasks describes other methods. (LLM, conf=0.82)
Experimental — accuracy not yet measuredEarlier methods evaluate code quality by generating test cases, but this suffers from inaccurate test generation, inability to handle special judges, and failure to assess non-code parts of responses.
“Previous works (Chen et al., 2023;Zhang et al., 2024a) rely on generating test cases to evaluate code quality.However, this method has issues with inaccurate test case generation (Jain et al., 2024), inability to handle special judge (Quan et al., 2025), and failure to evaluate non-code content in response such as code comments and explanations.”
From RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation (2025) · cites this paper as Chen et al., 2023
The citation to Chen et al. is attached to the claim that earlier work generates test cases to evaluate code quality. The paper’s “Execution-based code selection” section describes using unit tests supplied in the problem description to check code correctness, not generating those tests. The later limitations are attached to other citations, so they are not assessed as claims attributed to Chen et al. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe belief that LLMs can self-critique and improve their own solutions in an iterative fashion rests on the assumption that verification of correctness should be easier than generation.
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [5]
The Abstract and “Self-debugging with explanation” section support the premise that LLMs can iteratively identify and correct errors in their own code. They do not say that this ability rests on verification being easier than generation, or invoke a computational-complexity argument; searches for that comparison found no supporting passage. (LLM, conf=0.82)
Experimental — accuracy not yet measuredRecent research shows that LLMs such as ChatGPT demonstrate preliminary code debugging capabilities.
“Meanwhile, recent research shows that LLMs such as ChatGPT demonstrate preliminary code debugging capabilities (Chen et al., 2023b,a).”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Chen et al., 2023b
ChatGPT has shown preliminary capability in error detection and correction during code generation.
“Since ChatGPT has shown preliminary capability in error detection and correction during code generation (Shinn et al., 2023;Chen et al., 2023b;Kim et al., 2023;Nair et al., 2023;Madaan et al., 2023), we use Chat-GPT to perform multiple rounds of self-debugging and investigate the impact on DSR.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Chen et al., 2023b
The SELF-DEBUGGING approach endows LLMs with the ability to debug their output, reinforcing continual refinement in generated code.
“In terms of optimization, the SELF-DEBUGGING approach [14] leads the post-code generation phase.It endows LLMs with the ability to debug their output, reinforcing the theme of continual refinement in the generated code.”
From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [14]
The ability of LLMs to self-debug has been demonstrated effectively in recent research.
“The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023;Chen et al., 2023c;Lu et al., 2023;Kim et al., 2023).”
From Large language models as tool makers (2023) · cites this paper as Chen et al., 2023c
Various refinement strategies use external feedback from knowledge bases, code interpreters, tool outputs, or separately trained reward models.
“Various refinement strategies use external feedback from knowledge bases (Peng et al., 2023;Chern et al., 2023), code interpreters (Chen et al., 2023;Zhang et al., 2023), tool outputs (Gou et al., 2024;Schick et al., 2023), or separately trained reward models (Akyürek et al., 2023).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Chen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Conflicting sampling temperatures for initial translation
MinorThe experiments section specifies temperature 0.7 when sampling multiple initial programs, while Appendix F.1 specifies 0.8 for generating initial Python translations. Both describe the initial code-generation setting for TransCoder.
“When sampling multiple programs for a problem, we set temperature τ = 0.7, then we perform execution-based selection described in Section 2.”
“The prompt for generating the initial Python translation is taken from (Chowdhery et al., 2022).We set temperature τ = 0.8 as in (Chowdhery et al., 2022).”
Cites
Marsden, W. (2012) · doi:10.1017/cbo9781139207249.009 · Cited by 170,911 · cited 2×
“In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks [8, 13, 38, 67, 57].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1× · see citing works
“We compare SELF-DEBUGGING to Graphix-T5 [31], T5-3B + Syn data [66] and T5-3B + N-best Reranking [64], which are adapted from the T5 architecture [42] and specially trained for text-to-SQL generation.”
MethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 3× · see citing works
“For example, chain-of-thought prompting asks the model to produce intermediate reasoning steps in natural language, which can be accomplished either with few-shot exemplars (Wei et al., 2022) or in a zero-shot manner (Kojima et al.”
BackgroundSelf-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 5× · see citing works
“One popular technique is reinforcement learning from human feedback (RLHF) (Ziegler et al., 2019; Ouyang et al., 2022), and RLHF-trained models have demonstrated the ability to avoid harmful outputs when instructed to do so in the prompt (Ganguli et al., 2023).”
MethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 5× · see citing works
“The prompt for generating the initial Python translation is taken from (Chowdhery et al., 2022).”
BackgroundMethodInfluentialSelf-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 2× · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 10× · see citing works
“, 2023), and performing selection based on the consensus on unit test execution outputs among samples (Chen et al., 2019; 2021b; Roziere et al., 2022; Shi et al., 2022; Li et al., 2022; Chen et al., 2023b).”
BackgroundMethodInfluentialSelf-citationYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · cited 6× · see citing works
“Recent years have witnessed rapid progress in deep neural networks for code generation [14, 11, 63, 44].”
BackgroundMethodInfluentialDecomposed Prompting: A Modular Approach for Solving Complex TasksPreprint
Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., & Sabharwal, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.02406 · Cited by 839 · cited 3× · see citing works
“For example, decomposed prompting (Khot et al., 2022) delegates subproblems to other LLM instances that are prompted to specialize in specific tasks.”
BackgroundSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1× · see citing works
“Prior works demonstrate that decoding multiple samples can significantly improve the performance of large language models (Wang et al., 2023; Shi et al., 2022).”
BackgroundSelf-citationPICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
Scholak, T., Schucher, N., & Bahdanau, D. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.779 · Cited by 788 · cited 1× · see citing works
“While models designed and trained for specialized domains have achieved impressive performance in various applications such as text-to-code generation (Li et al., 2023a; Wang et al., 2020; Scholak et al., 2021; Dong & Lapata, 2016; Iyer et al., 2018) and code translation (Chen et al.”
MethodLanguage to Logical Form with Neural Attention
Dong, L., & Lapata, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1004 · Cited by 700 · cited 1× · see citing works
“have achieved impressive performance in various applications such as text-to-code generation [31, 52, 48, 15, 26] and code translation [10, 44, 45], latest work on large language models demonstrate that a single pretrained model can achieve the state-of-the-art performance across a wide variety of coding tasks without…”
BackgroundMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 1× · see citing works
“Besides input-output demonstrations, we can optionally add an instruction in the prompt to provide a high-level task description [41, 46, 51].”
MethodRAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Wang, B., Shin, R., Liu, X., Polozov, O., & Richardson, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.677 · Cited by 606 · cited 1× · see citing works
“While models designed and trained for specialized domains have achieved impressive performance in various applications such as text-to-code generation (Li et al., 2023a; Wang et al., 2020; Scholak et al., 2021; Dong & Lapata, 2016; Iyer et al., 2018) and code translation (Chen et al.”
MethodCodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Gotmare, A. D., Hoi, S., Le, H., Savarese, S., & Wang, Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2207.01780 · Cited by 605 · cited 3× · see citing works
“Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs (Gupta et al., 2017; Yasunaga & Liang, 2021; Gupta et al., 2020; Wang et al., 2018; Le et al., 2022).”
BackgroundA systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 2× · see citing works
“In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks (Chen et al., 2021a; Nijkamp et al., 2023; Zheng et al., 2023; Xu et al., 2022).”
BackgroundDeepFix: Fixing Common C Language Errors by Deep Learning
Gupta, R., Pal, S., Kanade, A., & Shevade, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10742 · Cited by 426 · cited 2× · see citing works
“Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs (Gupta et al., 2017; Yasunaga & Liang, 2021; Gupta et al., 2020; Wang et al., 2018; Le et al., 2022).”
BackgroundMethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 7× · see citing works
“Specifically, we evaluate on the MBPP test set (Austin et al., 2021), which contains 500 Python problems with text descriptions, where each problem has 3 unit tests.”
BackgroundMethodInfluentialLeast-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · cited 3× · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundSelf-citationLearning a Neural Semantic Parser from User Feedback
Iyer, S. V., Konstas, I., Cheung, A., Krishnamurthy, J., & Zettlemoyer, L. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1089 · Cited by 343 · cited 1× · see citing works
“, 2023), and fix generated programs based on human feedback (Iyer et al., 2017; Elgohary et al., 2020; Chen et al., 2023a; Le et al., 2022).”
BackgroundCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 3× · see citing works
“In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks (Chen et al., 2021a; Nijkamp et al., 2023; Zheng et al., 2023; Xu et al., 2022).”
BackgroundBai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · cited 5× · see citing works
“Recent works have shown the great promise of RLHF-trained models to generate critiques with prompting, which reduces harmful model outputs (Bai et al., 2022; Ganguli et al., 2023) and improves the performance on some reasoning tasks (Shinn et al.”
BackgroundMethodInfluentialCodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 263 · cited 2× · see citing works
“In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks (Chen et al., 2021a; Nijkamp et al., 2023; Zheng et al., 2023; Xu et al., 2022).”
BackgroundStarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL ParsingPreprint
Li, J., Hui, B., Cheng, R., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2301.07507 · Cited by 185 · cited 2× · see citing works
“We compare SELF-DEBUGGING to Graphix-T5 [31], T5-3B + Syn data [66] and T5-3B + N-best Reranking [64], which are adapted from the T5 architecture [42] and specially trained for text-to-SQL generation.”
BackgroundMethodMapping Language to Code in Programmatic Context
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1192 · Cited by 180 · see citing works
Tao, Y., Zhang, R., Er, H., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1204 · Cited by 161 · cited 1× · see citing works
“, 2023a), interact with users for multi-turn code generation (Yu et al., 2019; Yao et al., 2019; Nijkamp et al., 2023), and fix generated programs based on human feedback (Iyer et al.”
BackgroundTree-to-tree neural networks for program translation
Chen, X., Liu, C., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.03691 · Cited by 157 · cited 1× · see citing works
“, 2018) and code translation (Chen et al., 2018; Roziere et al., 2020; 2022), latest work on large language models demonstrate that a single pretrained model can achieve the state-of-the-art performance across a wide variety of coding tasks without specialized finetuning (Chen et al.”
BackgroundSelf-citationPAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155 · cited 1× · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundDynamic Neural Program Embedding for Program RepairPreprint
Wang, K., Singh, R., & Su, Z.-D. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.07163 · Cited by 145 · cited 4× · see citing works
“Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs (Gupta et al., 2017; Yasunaga & Liang, 2021; Gupta et al., 2020; Wang et al., 2018; Le et al., 2022).”
BackgroundMethodInfluentialShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 1× · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundCoder Reviewer Reranking for Code GenerationPreprint
Zhang, T., Yu, T., Hashimoto, T. B., Lewis, M., Yih, W.-T., Fried, D., & Wang, S. I. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2211.16490 · Cited by 144 · cited 8× · see citing works
“, 2022), while other works design reranking schemes to improve the performance (Zhang et al., 2022; Ni et al., 2023; Yin & Neubig, 2019; Zeng et al., 2022).”
BackgroundMethodInfluentialMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 1× · see citing works
“Some code generation tasks are accompanied with unit tests to specify the program execution behavior (Chen et al., 2021a; Austin et al., 2021; Li et al., 2022; Hendrycks et al., 2021).”
BackgroundA data-driven approach for learning to control computersPreprint
Humphreys, P., Raposo, D., Pohlen, T., et al. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2202.08137 · Cited by 133 · cited 1× · see citing works
“, 2023) demonstrates a related method combined with state and agent grounding that allows a large language model to accomplish computer control tasks in the MiniWoB++ benchmark (Humphreys et al., 2022), and improves zero-shot performance on several math and commonsense reasoning benchmarks.”
BackgroundRobustFill: Neural Program Learning under Noisy I/OPreprint
Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A., & Kohli, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07469 · Cited by 127 · cited 2× · see citing works
“Recent years have witnessed rapid progress in deep neural networks for code generation (Devlin et al., 2017; Chen et al., 2019; Yu et al., 2018; Roziere et al., 2020).”
BackgroundChen, W., Ma, X., Wang, X., & Cohen, W. W. (2022) · arXiv · doi:10.48550/arxiv.2211.12588 · Cited by 125 · cited 1× · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundLanguage Models can Solve Computer Tasks
Kim, G., Baldi, P., & McAleer, S. (2023) · arXiv · doi:10.48550/arxiv.2303.17491 · Cited by 101 · cited 5× · see citing works
“Recent works have shown the great promise of RLHF-trained models to generate critiques with prompting, which reduces harmful model outputs [3, 18] and improves the performance on some reasoning tasks [50, 35, 28, 36].”
BackgroundMethodInfluentialImproving Code Generation by Training with Natural Language FeedbackPreprint
Chen, A. (2023) · arXiv.org · doi:10.48550/arxiv.2303.16749 · Cited by 98 · cited 9× · see citing works
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them [6, 2], SELF-DEBUGGING teaches the model to identify the implementation errors via code explanation.”
BackgroundMethodInfluentialLeveraging Grammar and Reinforcement Learning for Neural Program SynthesisPreprint
Bunel, R., Hausknecht, M., Devlin, J., Singh, R., & Kohli, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04276 · Cited by 94 · cited 1× · see citing works
“, 2022), programming by examples (Devlin et al., 2017; Bunel et al., 2018; Chen et al., 2019), and code translation (Roziere et al.”
BackgroundCodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · cited 4× · see citing works
“Feedback: With the above function, square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100].”
BackgroundInfluentialGraph-based, Self-Supervised Program Repair from Diagnostic Feedback
Yasunaga, M., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10636 · Cited by 81 · see citing works
DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving AgentsPreprint
Nair, V., Schumacher, E., Tso, G., et al. (2023) · Clinical Natural Language Processing Workshop · doi:10.48550/arxiv.2303.17071 · Cited by 75 · cited 2× · see citing works
“, 2023) and improves the performance on some reasoning tasks (Shinn et al., 2023; Madaan et al., 2023b; Kim et al., 2023; Nair et al., 2023).”
BackgroundMethodNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 65 · see citing works
Reranking for Neural Semantic Parsing
Yin, P., & Neubig, G. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1447 · Cited by 62 · cited 1× · see citing works
“One line of work selects the final prediction using the majority vote of execution results [11, 32, 49], while other works design reranking schemes to improve the performance [65, 37, 61, 64].”
BackgroundModel-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
Yao, Z., Su, Y., Sun, H., & Yih, W.-T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1547 · Cited by 61 · cited 1× · see citing works
“, 2023a), interact with users for multi-turn code generation (Yu et al., 2019; Yao et al., 2019; Nijkamp et al., 2023), and fix generated programs based on human feedback (Iyer et al.”
BackgroundUnsupervised translation of programming languages
Roziere, B. (2026) · doi:10.70675/416f6582z4576z444eza710z5e930d59bf97 · Cited by 57 · cited 4× · see citing works
“On both TransCoder for code translation (Roziere et al., 2020) and MBPP for text-to-Python generation (Austin et al.”
BackgroundMethodInfluentialEvaluating the Text-to-SQL Capabilities of Large Language Models
Rajkumar, N., Li, R., & Bahdanau, D. (2022) · arXiv · doi:10.48550/arxiv.2204.00498 · Cited by 56 · cited 1× · see citing works
“Following prior work evaluating large language models for text-to-SQL generation (Rajkumar et al., 2022; Shi et al., 2022; Ni et al., 2023), we evaluate SELF-DEBUGGING on the development set of the Spider benchmark (Yu et al.”
MethodSpeak to your Parser: Interactive Text-to-SQL with Natural Language Feedback
Elgohary, A., Hosseini, S., & Awadallah, A. H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.187 · Cited by 56 · cited 4× · see citing works
“Feedback: With the above function, square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100].”
BackgroundMethodInfluentialImportance of Synthesizing High-quality Data for Text-to-SQL ParsingPreprint
Hu, Y., Zhao, Y., Jiang, J., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.08785 · Cited by 55 · cited 1× · see citing works
“We compare SELF-DEBUGGING to Graphix-T5 [31], T5-3B + Syn data [66] and T5-3B + N-best Reranking [64], which are adapted from the T5 architecture [42] and specially trained for text-to-SQL generation.”
MethodThe Capacity for Moral Self-Correction in Large Language Models
Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernandez, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07459 · Cited by 53 · cited 4× · see citing works
“Despite recent promising progress showing that large language models can generate critiques to avoid harmful model outputs (Ganguli et al., 2023; Bai et al., 2022) and improve their performance on some natural language tasks (Shinn et al.”
BackgroundInfluentialSelf-critiquing models for assisting human evaluators
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., & Leike, J. (2022) · arXiv · doi:10.48550/arxiv.2206.05802 · Cited by 50 · cited 1× · see citing works
“reasoning tasks [50, 28, 47], prior work has yet to show the effectiveness of model-generated feedback on code generation [6].”
BackgroundNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49 · cited 12×
“Following prior work evaluating large language models for text-to-SQL generation (Rajkumar et al., 2022; Shi et al., 2022; Ni et al., 2023), we evaluate SELF-DEBUGGING on the development set of the Spider benchmark (Yu et al.”
BackgroundMethodInfluentialLEVER: Learning to Verify Language-to-Code Generation with Execution
Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-T., Wang, S. I., & Lin, X. V. (2023) · arXiv · doi:10.48550/arxiv.2302.08468 · Cited by 46 · cited 10× · see citing works
“Following prior work evaluating large language models for text-to-SQL generation (Rajkumar et al., 2022; Shi et al., 2022; Ni et al., 2023), we evaluate SELF-DEBUGGING on the development set of the Spider benchmark (Yu et al.”
BackgroundMethodInfluentialChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 1× · see citing works
“Besides input-output demonstrations, we can optionally add an instruction in the prompt to provide a high-level task description (Ouyang et al., 2022; Sanh et al., 2022; Suzgun et al., 2022).”
MethodSelf-citationBreak-It-Fix-It: Unsupervised Learning for Program RepairPreprint
Yasunaga, M., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06600 · Cited by 43 · cited 5× · see citing works
“Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs [21, 60, 20, 53].”
BackgroundMethodInfluentialGenerating Sequences by Learning to Self-Correct
Welleck, S., Lu, X., West, P., Brahman, F., Shen, T., Khashabi, D., & Choi, Y. (2022) · arXiv · doi:10.48550/arxiv.2211.00053 · Cited by 41 · cited 1× · see citing works
“Another line of work trains a language model to refine the initial model outputs based on external feedback on prediction quality (Welleck et al., 2023; Liu et al., 2023), which improves the performance on several natural language and reasoning tasks.”
BackgroundLeveraging automated unit tests for unsupervised code translation
Roziere, B., Zhang, J. M., Charton, F., Harman, M., Synnaeve, G., & Lample, G. (2021) · arXiv · doi:10.48550/arxiv.2110.06773 · Cited by 39 · cited 1× · see citing works
“, 2023), and performing selection based on the consensus on unit test execution outputs among samples (Chen et al., 2019; 2021b; Roziere et al., 2022; Shi et al., 2022; Li et al., 2022; Chen et al., 2023b).”
BackgroundMulti-lingual Evaluation of Code Generation Models
Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14868 · Cited by 38 · cited 5× · see citing works
“def count_ways(n): A = [0] * (n + 1) B = [0] * (n + 1) A[0] = 1 A[1] = 0”
BackgroundInfluentialChain of Hindsight Aligns Language Models with Feedback
Liu, H., Sferrazza, C., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2302.02676 · Cited by 30 · cited 1× · see citing works
“Another line of work trains a language model to refine the initial model outputs based on external feedback on prediction quality (Welleck et al., 2023; Liu et al., 2023), which improves the performance on several natural language and reasoning tasks.”
BackgroundTeaching Algorithmic Reasoning via In-context LearningPreprint
Zhou, H., Nova, A., Larochelle, H., Courville, A. C., Neyshabur, B., & Sedghi, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09066 · Cited by 24 · cited 1× · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundPretraining Language Models with Human Preferences
Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., & Perez, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08582 · Cited by 23 · cited 1× · see citing works
“Training with feedback to improve the outputs of large language models, both in terms of correctness and alignment with human preferences, is an active research direction nowadays (Ziegler et al., 2019; Korbak et al., 2023; Ganguli et al., 2023; Bai et al., 2022).”
BackgroundLearning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 23 · cited 1× · see citing works
“For code generation, a number of works have trained models to perform code optimization [34], interact with users for multi-turn code generation [62, 58, 38], and fix generated programs based on human feedback [25, 16, 6].”
MethodN-Best Hypotheses Reranking for Text-to-SQL Systems
Zeng, L., Parthasarathi, S. H. K., & Hakkani-Tur, D. Z. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023434 · Cited by 20 · cited 6× · see citing works
“The assertion is "assert square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]".”
BackgroundMethodInfluentialSynthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis
Gupta, K., Christensen, P. E., Chen, X., & Song, D. (2020) · arXiv · doi:10.48550/arxiv.2007.08095 · Cited by 18 · cited 4× · see citing works
“Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs (Gupta et al., 2017; Yasunaga & Liang, 2021; Gupta et al., 2020; Wang et al., 2018; Le et al., 2022).”
BackgroundMethodInfluentialSelf-citation(2003) · Science · doi:10.1126/science.299.5604.185b · Cited by 14 · cited 5×
“Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs [21, 60, 20, 53].”
BackgroundMethodInfluentialN-Best Hypotheses Reranking for Text-To-SQL SystemsPreprint
Zeng, L., Parthasarathi, S. H. K., & Hakkani-Tür, D. Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10668 · Cited by 3 · see citing works
Latent Execution for Neural Program Synthesis
Chen, X., Song, D., & Tian, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.00101 · Cited by 2 · cited 1× · see citing works
“Therefore, several approaches have been proposed to leverage code execution to choose the final prediction from multiple candidates, such as utilizing or training a language model for reranking [65, 37], and performing selection based on the consensus on unit test execution outputs among samples [11, 12, 45, 49, 32, 7…”
BackgroundSelf-citationMeasuring The Impact Of Programming Language Distribution
Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · cited 4×
“### Task Start ### # These are the assertions for your function: assert small_nnum([10, 20, 50, 70, 90, 20, 50, 40, 60, 80, 100],2)==[10,20]”
BackgroundInfluentialBMJ Publishing Group Ltd, & Royal College of Psychiatrists and British Psychological Society (2008) · Evidence-Based Mental Health · doi:10.1136/ebmh.11.4.102 · cited 26×
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them [6, 2], SELF-DEBUGGING teaches the model to identify the implementation errors via code explanation.”
BackgroundMethodInfluentialOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Self-citation
Flagged first, then by citations
1 of the 75 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In Interna- tional Conference on Learning Representations, 2019.
- Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, and Jishen Zhao. Coda: An end-to-end neural program decompiler. In NeurIPS, 2019.
Cited by
Sorted by citations
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · see citing works
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573 · see citing works
“Li et al. [87] employs the LLM in finding the failure-inducing test cases.”
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544 · see citing works
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
“Chen et al. [150] introduced SELF-DEBUGGING, which teaches an LLM to debug its predicted code via few-shot learning, SELF-DEBUGGING reports baseline accuracy improvements of up to 12%.”
MethodEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315 · see citing works
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087 · see citing works
“Self-Debugging [6] employs a debugging component that is used to improve existing implementations given feedback from a code execution environment.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · see citing works
“Self-debugging [Chen et al., 2023] teaches LLM to perform rubber duck debugging after code generation.”
Automatic Prompt Optimization with “Gradient Descent” and Beam Search
Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C., & Zeng, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.494 · Cited by 191 · see citing works
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151 · see citing works
Self-citationLogic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“Recent works (Chen et al., 2023; Madaan et al., 2023) have adopted similar ideas to improve code generation, by teaching LLMs to debug their predicted programs via fewshot demonstrations.”
BackgroundLLM for SoC Security: A Paradigm Shift
Saha, D., Tarek, S., Yahyaei, K., Saha, S. K., Zhou, J., Tehranipoor, M. M., & Farahmandi, F. (2024) · IEEE Access · doi:10.1109/access.2024.3427369 · Cited by 113 · see citing works
“This has led to the emergence of several prompting techniques [115], [116], [117], [118], [119], [120], [121], [122], [123], [124], [125], [126], [127], [128], [129].”
BackgroundMethodDesigning Heterogeneous LLM Agents for Financial Sentiment Analysis
Xing, F. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3688399 · Cited by 88 · see citing works
“LLM collaboration is also more investigated on many general natural language processing tasks including sentiment analysis, but their applicability on FSA lacks direct evidence.”
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86 · see citing works
“…et al., 2023; Qin et al., 2023), while also showcasing surprising abilities of language understanding (Wei et al., 2022a; Begus et al., 2023), generation (Pu and Demberg, 2023; Lin and Chen, 2023; Lyu et al., 2023a), and reasoning (Wei et al., 2022b; Kojima et al., 2022; Dasgupta et al., 2022).”
BackgroundKamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“…et al., 2023; Yao et al., 2022b; Parisi et al., 2022; Tang et al., 2023b), and websites (Shi et al., 2017; Nakano et al., 2021; Yao et al., 2022a; Zhou et al., 2023b; Gur et al., 2023; Deng et al., 2023) as well as general code execution (Yang et al., 2023; Le et al., 2022; Ni et al., 2023).”
BackgroundGeneralized Planning in PDDL Domains with Pretrained Large Language Models
Silver, T., Dan, S., Srinivas, K., Tenenbaum, J. B., Kaelbling, L. P., & Katz, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.30006 · Cited by 90 · see citing works
LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 77 · see citing works
“Chen et al. [63] propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.”
BackgroundOlausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57 · see citing works
“Second, Logic-LM employs a self-refinement strategy, which has shown promise across code generation and NLP tasks (Zhang et al., 2023b; Chen et al., 2023a; Peng et al., 2023; Madaan et al., 2023; Olausson et al., 2023) but which we do not consider here.”
MethodCRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
PURPLE: Making a Large Language Model a Better SQL Writer
Ren, T., Fan, Y., He, Z., Huang, R., Dai, J., Huang, C., Jing, Y., Zhang, K., Yang, Y., & Wang, X. S. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00009 · Cited by 65 · see citing works
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54 · see citing works
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 71 · see citing works
“Prior work shows that LLMs can improve their responses via feedback (Shinn et al., 2023; Paul et al., 2023; Madaan et al., 2023; Ganguli et al., 2023; Ma et al., 2023; Chen et al., 2023b; Peng et al., 2023; Kim et al., 2023; Du et al., 2023; Liang et al., 2023; Chen et al., 2023a; Pan et al., 2023).”
BackgroundClover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43 · see citing works
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 57 · see citing works
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42 · see citing works
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
Self-citation(2023) · doi:10.18653/v1/2023.emnlp-main.364 · Cited by 38 · see citing works
Applying fine-tuned LLMs for reducing data needs in load profile analysis
Hu, Y., Kim, H., Ye, K., & Lü, N. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124666 · Cited by 34 · see citing works
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.337 · Cited by 31 · see citing works
RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
Akyürek, A. F., Akyürek, E., Kalyan, A., Clark, P. E., Wijaya, D. T., & Tandon, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.427 · Cited by 45 · see citing works
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28 · see citing works
Self-citation<b>JARVIS</b> -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511593 · Cited by 36 · see citing works
An Approach for Rapid Source Code Development Based on ChatGPT and Prompt Engineering
Li, Y., Shi, J., & Zhang, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3385682 · Cited by 27 · see citing works
“This section incorporates the Self-Debugging method [32] for optimizing the Prompt in Fig.”
MethodChang, H., & Li, T. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101755 · Cited by 35 · see citing works
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29 · see citing works
“We evaluate F AC T OOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023; Chen et al., 2023).”
MethodLLMs cannot find reasoning errors, but can correct them given the error location
Tyen, G., Mansoor, H., Carbune, V., Chen, P., & Mak, T. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.826 · Cited by 24 · see citing works
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 43 · see citing works
“While early models 3,4 were characterized by their limited scope and functional capabilities, primarily tailored for specific tasks like text classification and sentiment analysis, the landscape of natural language processing (NLP) underwent a transformative shift with the introduction of neural…”
Large Language Model Operations (LLMOps): Definition, Challenges, and Lifecycle Management
Díaz-de-Arcaya, J., López-de-Armentia, J., Miñón, R., Ojanguren, I. L., & Torre-Bastida, A. I. (2024) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech61897.2024.10612341 · Cited by 22 · see citing works
“Model Selection [6], [15], [21], [22], [23], [27], [25], [28], [29] S3 Adaptation [6], [14], [15], [21], [22], [23], [24], [27], [25], [28], [30], [31] S4 Evaluation [14], [15], [22], [23], [24], [27], [25], [28], [32], [33], [34] S5 Deployment [15], [21], [22], [23], [27], [25], [28], [35] S6…”
BackgroundAutomating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
Alkafaween, U., Al-Bluwi, I., & Denny, P. C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13100 · Cited by 23 · see citing works
Tang, H., Key, D., & Ellis, K. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2243 · Cited by 20 · see citing works
Selective Demonstrations for Cross-domain Text-to-SQL
Chang, S., & Fosler‐Lussier, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.944 · Cited by 22 · see citing works
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 23 · see citing works
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 25 · see citing works
“…methods [196], [197], [198], [199], uncertainty estimation [147], [200], [201], [202], [203], [204], [205], controlled decoding [206], [207], [208], [209], self-refinement [210], [211], [212], [213], [214], and leveraging external knowledge during inference [175], [215], [216], [217], [218], [219].”
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“This approach is problematic as (1) closed-source APIs keep changing and have unpredictable availability (Pozzobon et al., 2023; Chen et al., 2023a), (2) it relies on the assumption that a more capable model exists (3) it can reinforce model hallucination (Gudibande et al., 2023) and (4) ,…”
BackgroundExploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Dai, Y. Q., Lang, H., Zeng, K., Huang, F., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.351 · Cited by 20 · see citing works
Iterative Prompt Refinement for Mining Gene Relationships from ChatGPT
Chen, Y., Gao, J., Petruc, M., Hammer, R., Popescu, M., & Xu, D. H. (2024) · International Journal of Artificial Intelligence and Robotics Research · doi:10.1142/s2972335324500054 · Cited by 15 · see citing works
Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap
Staniek, M., Schumann, R., Züfle, M., & Riezler, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00654 · Cited by 15 · see citing works
“With the advent of large language models and in-context learning, more recent work focuses on prompt engineering the task (Sun et al., 2023; Chen et al., 2023; Pourreza and Rafiei, 2023).”
BackgroundSänger, M., Mecquenem, N. D., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., & Kosch, T. (2024) · GigaScience · doi:10.1093/gigascience/giae030 · Cited by 15 · see citing works
COFFE: A Code Efficiency Benchmark for Code Generation
Peng, Y., Wan, J., Li, Y., & Ren, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715727 · Cited by 15 · see citing works
“Researchers have proposed different approaches to make use of LLMs on code generation via prompting engineering [10, 58, 70, 74, 94], multiple-agent cooperation [29, 30, 32, 80, 87, 92], and retrieval augmentation [50, 67, 77, 90, 95].”
Can ChatGPT support software verification?
Janßen, C., Richter, C., & Wehrheim, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3_13 · Cited by 19 · see citing works
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12 · see citing works
“…coding capabilities of LLM without the need to adjust its parameters (Chen et al., 2022; Huang et al., 2023; Li et al., 2023a; Zhang et al., 2023c; Chen et al., 2023; Key et al., 2022) However, most of these works set up the evaluation in scenarios LLMs are familiar with, e.g., HumanEval (Chen et…”
Yao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., & Yu, H. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.538 · Cited by 12 · see citing works
Nguyen, M. T., Tran, K. T., Nguyen, N. V., & Vu, X.-S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.70 · Cited by 11 · see citing works
“Recent studies have explored various methods for fine-tuning LLMs with instruction data, enhancing their performance on specific tasks and domains (Koleva et al., 2022; Qiao et al., 2022; Li et al., 2023; Wu et al., 2023; Chen et al., 2023).”
Guo, C., Tian, Z., Tang, J., Li, S., & Wang, T. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103978 · Cited by 15 · see citing works
Luo, Q., Lin, Q., Xu, L., Wu, S., Mao, R., Wang, C., Feng, H., Huang, B., & Du, Z. (2026) · Big Earth Data · doi:10.1080/20964471.2026.2615511 · Cited by 11 · see citing works
Large Language Models Can Self-Correct with Key Condition Verification
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.714 · Cited by 10 · see citing works
“Other sources, such as compilers (Chen et al., 2024; Zeng et al., 2021) or search engines (Yu et al., 2023b) can provide external feedback.”
BackgroundMethodPlan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Liu, T., Guo, Q., Yang, Y., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.169 · Cited by 12 · see citing works
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
Chen, H., Saha, A., Hoi, S. C. H., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.417 · Cited by 9 · see citing works
Peng, Z., Kuai, X., Ke, S., Dong, X., & Guo, R. (2025) · Big Earth Data · doi:10.1080/20964471.2025.2483541 · Cited by 9 · see citing works
“Chen et al. (2023b) incorporated error messages as prompts for LLMs, enhancing accuracy through a self-debugging approach.”
ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8 · see citing works
“The effectiveness of causal reasoning pathways [11] is also crucial, ensuring that the output of LLMs is accurate but also transparent and verifiable.”
BackgroundLLM Critics Help Catch LLM Bugs
McAleese, N., Pokorny, R. M., Uribe, J. F. C., Nitishinskaya, E., Trebacz, M., & Leike, J. (2024) · arXiv · doi:10.48550/arxiv.2407.00215 · Cited by 10 · see citing works
“Improvements from self-correction and closely related methods have been claimed for harmlessness [1], factuality [24], computer control [14] and moral reasoning [5], as well as in other domains [20, 4, 7].”
BackgroundA Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8 · see citing works
MeshAgent: enabling reliable network management with large language models
Zhou, Y., Hsieh, K., Mani, S. K., Kandula, S., & Liu, Z. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771567 · Cited by 7 · see citing works
“Even advanced prompting fails on networking queries that require precise multi-step reasoning.”
Exploring Large Language Models in Information Systems: A Survey
Bianchini, F., & Marinacci, M. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_5 · Cited by 6 · see citing works
On the Challenges of Using Large Language Models for NCL Code Generation
Moraes, D. D. S., da Costa, P. B., Busson, A. J. G., Boaro, J. M. C., Neto, C. D. S. S., & Colcher, S. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia_estendido.2023.236175 · Cited by 6 · see citing works
“One of the main challenges is the creation or use of a sufficiently extensive dataset for training the LLM or fine-tuning a pre-trained model, as done in [8, 9, 14, 19, 22].”
BackgroundMethodTrojanSQL: SQL Injection against Natural Language Interface to Database
Zhang, J., Zhou, Y., Hui, B., Liu, Y., Li, Z., & Hu, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.264 · Cited by 6 · see citing works
“The emergence of powerful large language models (LLMs) has recently enabled the development of highly effective parsers with minimal demostra-tion examples (Chen et al., 2023), indicating the potential for LLM-based parsers to serve as novel interfaces for databases (Li et al., 2023).”
BackgroundMethodFully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 11 · see citing works
“Several studies explore an iterative approach to program synthesis in a manner similar to SEIDR [Chen et al., 2023; Shinn et al., 2023; Xia and Zhang, 2023].”
Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing
Yan, H., Srivastava, S., Tai, Y., Wang, S. I., Yih, W.-T., & Yao, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.177 · Cited by 7 · see citing works
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5 · see citing works
“The repair process can be aided with different artifacts, e.g., LLM-generated feedback on the code [27], error information produced by code execution [12, 14, 18].”
Fundamental Approaches to Software Engineering
Beyer, D., & Cavalcanti, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3 · Cited by 5 · see citing works
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1354 · Cited by 5 · see citing works
Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
Yang, Z., Zhang, Y., Wang, Y., Xu, Z., Lin, J., & Sui, Z. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.203 · Cited by 5 · see citing works
Flows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 14 · see citing works
Large Language Model Guided Self-Debugging Code GenerationPreprint
Adnan, M., Kuhn, C. C. N., & Xu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5396508 · Cited by 6 · see citing works
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4 · see citing works
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Cheng, C.-A., Nie, A., & Swaminathan, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2287 · Cited by 4 · see citing works
The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models
Ruiz, F. V., Hort, M., & Moonen, L. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756966 · Cited by 4 · see citing works
“Successive feedback loops in the generation process have been shown to improve outcomes [3, 8].”
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-TalkPreprint
Ulmer, D., Mansimov, E., Lin, K., et al. (2024) · IT University Of Copenhagen (IT University of Copenhagen) · doi:10.48550/arxiv.2401.05033 · Cited by 3 · see citing works
“…improvement in responses has been observed through the use of an external critique model (Paul et al., 2023), or through self-critiquing for a diverse set of tasks such as information extraction (Gero et al., 2023), code generation (Chen et al., 2023) and summarization (Saunders et al., 2022).”
MethodOn the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 3 · see citing works
“These take many forms: common search algorithms with the LLM cast in the role of heuristic [37, 11], approaches which reduce error rates by enforcing various consistency conditions [8, 6, 17], and direct LLM self-critique [27, 35, 5, 15, 21, 38].”
MethodMind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3 · see citing works
Self-citationEnhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
Di, Y., & Zhang, T. (2025) · arXiv · doi:10.48550/arxiv.2505.07768 · Cited by 3 · see citing works
TraffiCoT-R: A framework for advanced spatio-temporal reasoning in large language models
Alsahfi, T., & Qasim, K. U. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.05.027 · Cited by 2 · see citing works
Exploring Large Language Models for Text-to-SQL Error Correction with LoRA Fine-Tuning
Jiang, G., & Ma, C. (2025) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace65325.2025.11018984 · Cited by 2 · see citing works
“Self-debugging [6] leverages the concept of rubber duck debugging [13] by providing additional column predictions and explanation for the original question and SQL to generate error feedback, guiding the model in error correction.”
SGEU: enhancing LLM reasoning via backward exemplar generation and verification
Wang, Z., Zhou, X., Yang, Y., Ma, B., Wang, L., & Dong, R. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06529-8 · Cited by 1 · see citing works
A Survey of Natural Language-Based Editing of Low-Code Applications Using Large Language Models
Gorissen, S. C., Sauer, S., & Beckmann, W. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64576-1_15 · Cited by 2 · see citing works
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 2 · see citing works
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
“These works are often used to improve the capabilities of LLM agents, which are typically evaluated on simulated (Ruan et al., 2023) and real-world (Zhou et al., 2023b) environments.”
BackgroundSelf-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4 · see citing works
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2024) · arXiv · doi:10.48550/arxiv.2410.04444 · Cited by 2 · see citing works
“Some agents are de-LLM signed to iterate over a fixed routine consisting of a list of fixed modules, while some of the modules are capable of taking self-or environment feedback to refine their actions (Chen et al., 2023b; Qu et al., 2024a; Tang et al., 2025).”
Chain-of-Program Prompting with Open-Source Large Language Models for Text-to-SQL
Xu, B., Li, S., Wu, Y., Wei, S., Du, M., Wang, H., & Song, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650552 · Cited by 2 · see citing works
“Researchers continuously propose better context learning prompts to help models parse questions more effectively [6]–[8], [15], [16], leading to superior performance.”
BackgroundMethodRunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 11 · see citing works
Regression Accumulation in Multi-turn LLM Programming Conversations
Huang, A., Ma, L., Tahir, A., Zhang, Q., Xiao, L., & Xiao, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834357 · Cited by 1 · see citing works
“A separate line of work investigates iterative repair through self-debugging, execution feedback, and learned self-correction [9, 15, 25, 28, 41].”
Wang, R., Todd, G., Yuan, X., Xiao, Z., Côté, M.-A., & Jansen, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.830 · Cited by 2 · see citing works
Steve Jobs: Pioneering AI in Software Engineering
Priyadharasini, M., Sriram, S., Sudhar Aathith, T., et al. (2024) · International Research Journal on Advanced Engineering Hub (IRJAEH) · doi:10.47392/irjaeh.2024.0116 · Cited by 1 · see citing works
TT-SI: Self-Improving LLM Agents with Test-Time Training
Acikgoz, E. C., Qian, C., Ji, H., Hakkani-Tür, D., & Tur, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.462 · Cited by 1 · see citing works
“Recent studies suggest that LLMs can self-improve their capabilities (Huang et al., 2023; Wang et al., 2023; Chen et al., 2024a; Pang et al., 2024; Yuan et al., 2024; Shafayat et al., 2025; Huang et al., 2025), i.e., they can refine their own output distribution using internal signals derived from…”
- Authors
- Chen, X., Lin, M., Schärli, N., & Zhou, D.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.05128
- OpenAlex
- W4365205411
- Cited by
- 77
- References
- 75
- Citations by year
- 2026: 11 · 2025: 15 · 2024: 29 · 2023: 22
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, program repair, code explanation, text-to-SQL, sample efficiency, unit testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
