The DOI matches its DataCite record; 77 of the 77 references it cites were found in a registry or an index; 6 of the 16 sources its claims cite were checked for support.
Teaching Large Language Models to Self-Debug
Open accessAbstract
Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
SELF-DEBUGGING teaches LLMs to debug their own generated code via few-shot prompting, using execution results and natural-language code explanations without additional training. It improves state-of-the-art accuracy on text-to-SQL, code translation, and text-to-Python benchmarks, and notably improves sample efficiency.
Key findings
- On Spider text-to-SQL, where no unit tests are available, SELF-DEBUGGING with code explanation improves the Codex baseline from 81.3 to 84.1, and improves accuracy on the hardest problems by 9%.
- On TransCoder C++-to-Python and MBPP text-to-Python, using unit tests plus code explanation improves baseline accuracy by up to 12%; e.g., Codex on TransCoder improves from 80.4 to 92.5 with UT + Expl.
- SELF-DEBUGGING improves sample efficiency: on Spider, applying it to greedy decoding matches the baseline using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline using 32 samples.
- Error analysis shows Spider fixes often address wrong WHERE conditions (25.7%), missing DISTINCT (17.1%), and wrong JOIN clauses (14.3%); on TransCoder and MBPP, 60%–70% of successful fixes address output mismatch errors.
Methods
- The method uses few-shot prompting with a pretrained LLM, without finetuning, in iterative turns of Generation, Explanation, and Feedback; feedback can be simple, unit-test-based, code-explanation-based, or execution-trace-based.
- For multiple initial samples, the paper selects the program with the most frequent execution result among those without execution errors, filtering by unit tests when available; maximum debugging turns is 10, though successful debugging mostly ends within 3 turns.
- Evaluation uses Codex, GPT-3.5, GPT-4, and StarCoder on Spider development set, TransCoder C++-to-Python with 560 problems and 10 unit tests each, and MBPP with 500 Python problems and 3 unit tests each, with the first MBPP unit test included in the prompt.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: on MBPP, context length limits reduce the number of exemplars when adding code explanation (6 exemplars for simple/unit-test feedback vs. 3 for unit test + explanation), and code explanation is less significant when the initial predicted code is far from correct.
- Evident from the text: GPT-4 is tuned for zero-shot prompting and does not follow few-shot exemplars as well as Codex on Spider; evaluation is limited to specific benchmarks and C++-to-Python translation rather than broader languages or tasks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
95 Own result · 19 Own interpretation · 9 No basis given · 4 Partly supported · 4 Sources not found · 2 Own method · 1 Supported · 1 Supported so far
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“On the other hand, large language models have been shown to be capable of describing their generated problem solutions in both text (Wei et al., 2022;Kojima et al., 2022;Zhou et al., 2023) and code (Gao et al., 2022;Chen et al., 2022) formats.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Code Explanation feedback (Expl).
What the sources say
- Not checked — no full text of the sourcecites Wei et al., 2022 source
- Supportedcites Gao et al., 2022 source
The Gao et al. citation supports the code-format part of the sentence. The Introduction (Section 1) says PAL uses an LLM to generate programs as reasoning steps for natural-language problems; the Python interpreter then executes those steps to obtain the final answer. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that SELF-DEBUGGING consistently improves the performance.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.1 MAIN RESULTS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As will be shown in Section 5, it is insufficient for accurately predicting SQL correctness when the few-shot prompt only contains simple feedback.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 4.1 TEXT-TO-SQL GENERATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, on Spider, applying SELF-DEBUGGING to predictions generated with greedy decoding matches the baseline accuracy using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline accuracy using 32 samples.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.1 SELF-DEBUGGING IMPROVES THE SAMPLE EFFICIENCY
plainn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the Spider benchmark where there are no unit tests to verify the correctness of predictions, SELF-DEBUGGING with code explanation consistently improves the baseline by 2 -3%”
- Also stated in: § 1 INTRODUCTION · plain
“On the Spider benchmark (Yu et al., 2018) for text-to-SQL generation where there are no unit tests in the problem description, with code explanation, SELF-DEBUGGING consistently improves the baseline by 2 -3% with different numbers of initial programs”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we observe that the initial SQL queries generated by LLMs are usually not completely wrong, but they tend to make small mistakes when the questions require more complex SQL queries, e.g., missing a few WHERE conditions or SELECT columns.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, on extra hard problems, SELF-DEBUGGING increases the accuracy by 9%.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Without unit test execution, both models tend to be overconfident in their own initial predictions, while GPT-4 is better than GPT-3.5 on Python generation.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.2 IMPORTANCE OF CODE EXECUTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to Codex, few-shot prompting does not notably improve the SELF-DEBUGGING performance of GPT-3.5 and GPT-4, and thus their SELF-DEBUGGING procedures fully rely on their internal code knowledge.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.2 IMPORTANCE OF CODE EXECUTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Section 5.2.2, we further present an ablation study showing that leveraging code execution is crucial for enabling a performance leap with SELF-DEBUGGING.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 27
- The quote only partly states this claim. 9
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 1 INTRODUCTION
What the sources say
- Not matchedcites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Austin et al., 2021 source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior works demonstrate that decoding multiple samples can significantly improve the performance of large language models (Wang et al., 2023;Shi et al., 2022).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution-based code selection.
What the sources say
- Not matchedcites Wang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Shi et al., 2022 source
Section 4.3 (Primary Results; page unavailable) states that more samples improve performance for most selection methods and that MBR-EXEC significantly outperforms baselines without selection. The paper notes an exception: MALL performs worse with larger sample sizes. That exception is consistent with the claim that multiple samples *can* improve performance. (LLM, conf=0.91)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In particular, for code generation tasks, we can utilize code execution to select the final prediction (Chen et al., 2019;Li et al., 2022;Shi et al., 2022;Zhang et al., 2022;Ni et al., 2023).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution-based code selection.
What the sources say
- Not matchedcites Chen et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Li et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Shi et al., 2022 source
The Introduction describes sampling candidate programs and selecting a single prediction based on their execution results. Section 3 likewise defines the proposed approach as selecting the best candidate from code-model samples, directly supporting the claim. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work on code repair has demonstrated that training the repair model on execution traces improves the debugging performance (Wang et al., 2018;Gupta et al., 2020).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution trace feedback (Trace).
What the sources say
- Not matchedcites Wang et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Gupta et al., 2020 source
In Results (p. 6), the authors report that the debugger with TraceEmbed had the lowest repair error at every tested number of mutations, demonstrating that execution traces were helpful. The mutation-benchmark discussion (p. 7) also reports mostly better performance with trace embedding. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 2 PROMPTING FOR CODE GENERATION
What the sources say
- Supportedcites Brown et al., 2020 source
Section 2, Approach (p. 6) defines few-shot learning as giving the model several task demonstrations at inference time. It specifies that each demonstration pairs a context with a desired completion, after which the model is expected to complete a new context. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“and does not produce meaningful changes to the initial predicted SQL queries.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that on TransCoder and MBPP benchmarks where at least one unit test is available for SELF-DEBUGGING, simple feedback also utilizes the execution result to infer the code correctness even if the execution information is not presented in the feedback message, thus simple feedback still improves the model prediction for both applications.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Although GPT-4 is shown to be much stronger than previous GPT models on many tasks (Ope-nAI, 2023), on Spider, both its initial SQL generation and SELF-DEBUGGING performance are much worse than Codex.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite that the baseline performance of StarCoder is considerably worse than GPT models, SELF-DEBUGGING with unit test execution also offers a significant performance gain, i.e., 6% on MBPP with the execution trace feedback.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all tasks, models generally benefit from richer feedback for SELF-DEBUGGING, especially when the execution information is present in the feedback.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meanwhile, by leveraging feedback messages and reusing failed predictions, SELF-DEBUGGING notably improves sample efficiency, and can match or outperform baseline models that generate more than 10× candidate programs.”
- Also stated in: § 1 INTRODUCTION · plain
“Meanwhile, SELF-DEBUGGING improves sample efficiency, and can match or outperform baseline models that sample more than 10× predictions.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though empirically the successful debugging processes mostly end within 3 turns”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5 EXPERIMENTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar sample efficiency improvement on other benchmarks, and we defer the discussion to Appendix C.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.1 SELF-DEBUGGING IMPROVES THE SAMPLE EFFICIENCY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that typically one debugging turn is sufficient, and the accuracy improvement after one turn is within 0.1%.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.1 SELF-DEBUGGING IMPROVES THE SAMPLE EFFICIENCY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments, we will demonstrate that leveraging unit tests whenever available consistently boosts the SELF-DEBUGGING performance.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution-based code selection.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meanwhile, its self-debugging performance gain is on par with Codex: over 12% improvement on Transcoder, and 8% improvement on MBPP.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, we demonstrate that SELF-DEBUGGING can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong COUNT columns | 5.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 9: An example where SELF-DEBUGGING with unit test feedback fixes the code translation error, while the simple feedback fails.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > C.2 TEXT-TO-PYTHON GENERATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the translation with code explanation correctly checks for the end of a string.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > + Expl.
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing nested conditions | 5.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong WHERE conditions | 25.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong INTERSECT/UNION clauses | 8.6”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong column selection | 5.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“See Figure 11 in Appendix D for an example where SELF-DEBUGGING makes major changes to the initial code.”
- Also stated in: § Original Python code · plain
“Figure 11: An example on MBPP where the prediction after SELF-DEBUGGING is very different from the initial code.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > C.2 TEXT-TO-PYTHON GENERATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Original Python code · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 10: An example where adding code explanation results in the correct code translation, while the baseline prompt does not.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > + Expl.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong aggregate functions and keywords 5.8”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also performs close to GPT-3.5 on MBPP, and outperforms it for both Simple and UT.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“prior works suggest that such large language models are not yet capable of correcting code when lacking external feedback, such as unit tests or human instructions (Chen et al., 2023a).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 1 INTRODUCTION
What the sources say
- Not matchedcites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, text-to-SQL generation represents the scenario where no unit tests are available, where SELF-DEBUGGING enables the model to identify errors and justify code correctness by explaining the predicted code.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 4 APPLICATIONS In our evaluation, we showcase applications of SELF-DEBUGGING to different code generation domains: text-to-SQL generation, code translation, and text-to-Python generation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This debugging process is reminiscent of rubber duck debugging for human programmers, where explaining the code line-byline in natural language to a rubber duck significantly boosts debugging efficiency without expert guidance (Hunt & Thomas, 2000).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 1 INTRODUCTION
What the sources say
- Not matchedcites Hunt & Thomas, 2000
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since the unit tests are not available, it becomes more challenging for the model to infer the correctness of the predicted SQL queries.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 4.1 TEXT-TO-SQL GENERATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that both the execution trace and the line-by-line explanation come from model generation instead of code execution, thus the trace feedback does not require more information than the pure code explanation feedback; i.e., no access to intermediate execution states.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution trace feedback (Trace).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, we demonstrate that SELF-DEBUGGING empowers the model to perform rubber duck debugging, so that the model can identify and fix bugs without human instructions.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(Note that after this kind of feedback, we finish SELF-DEBUGGING and do not continue with”
§ Write Python function to complete the task and pass the assertion tests.### Task Start ### # These are the assertions for your function: assert count_ways(2) == 3 """ Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x n board.""" def count_ways(n): > Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, SELF-DEBUGGING teaches the large language model to notice prediction errors via code explanation in the style of rubber duck debugging, which does not require the feedback message to explicitly explain the implementation errors and how to fix them.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The new code prediction is used only if it also passes the unit test, otherwise the original prediction which passes will be used for evaluation.”
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
“The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.”
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
“The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.”
§ Write Python function to complete the task and pass the assertion tests.### Task Start ### # These are the assertions for your function: assert count_ways(2) == 3 """ Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x n board.""" def count_ways(n): > Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our work highlights the promise of improving the coding performance of large language models by teaching them to iteratively debug their own predictions, instead of requiring the model to generate the correct code from scratch.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we focus on code generation applications, and SELF-DEBUGGING demonstrates the effectiveness of both self-generated feedback and unit test feedback acquired by code execution.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, SELF-DEBUGGING enables the model to generate feedback messages on its own at test time, and does not require extra training.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper relies on this without support.
For complex programming tasks, generating the correct solution in one go is challenging.
“However, for complex programming tasks, generating the correct solution in one go becomes challenging”
“However, for many programming tasks, generating correct code with a single attempt is challenging.”
The paper relies on this without support.
Human programmers typically look into incorrect code and investigate execution results, then make changes to resolve implementation errors.
“Instead of completely discarding the incorrect code, humans typically look into the code and investigate execution results, then make changes to resolve the implementation errors.”
The paper relies on this without support.
There is no guarantee that code written on the first try by human programmers is always accurate.
“Intuitively, even for human programmers, there is no guarantee that the code written on the first try is always accurate.”
The paper relies on this without support.
Human programmers can usually identify bugs by describing code implementation and comparing it to the problem description without extra guidance.
“By describing the code implementation and comparing it to the problem description, human programmers are usually able to identify the bugs without extra guidance.”
The paper relies on this without support.
The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
“where the difficulty of each problem is annotated in the Spider benchmark based on the complexity of the ground truth SQL queries.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [7]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We evaluate FACTOOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023;Chen et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous works (Chen et al., 2023;Zhang et al., 2024a) rely on generating test cases to evaluate code quality.However, this method has issues with inaccurate test case generation (Jain et al., 2024), inability to handle special judge (Quan et al., 2025), and failure to evaluate non-code content in response such as code comments and explanations.”
From RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation (2025) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Meanwhile, recent research shows that LLMs such as ChatGPT demonstrate preliminary code debugging capabilities (Chen et al., 2023b,a).”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Chen et al., 2023b
The cited source says what the claim attributes to it.
In the citing paper
“Since ChatGPT has shown preliminary capability in error detection and correction during code generation (Shinn et al., 2023;Chen et al., 2023b;Kim et al., 2023;Nair et al., 2023;Madaan et al., 2023), we use Chat-GPT to perform multiple rounds of self-debugging and investigate the impact on DSR.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Chen et al., 2023b
The cited source says what the claim attributes to it.
In the citing paper
“In terms of optimization, the SELF-DEBUGGING approach [14] leads the post-code generation phase.It endows LLMs with the ability to debug their output, reinforcing the theme of continual refinement in the generated code.”
From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023;Chen et al., 2023c;Lu et al., 2023;Kim et al., 2023).”
From Large language models as tool makers (2023) · cites this paper as Chen et al., 2023c
The cited source says what the claim attributes to it.
In the citing paper
“Various refinement strategies use external feedback from knowledge bases (Peng et al., 2023;Chern et al., 2023), code interpreters (Chen et al., 2023;Zhang et al., 2023), tool outputs (Gou et al., 2024;Schick et al., 2023), or separately trained reward models (Akyürek et al., 2023).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Chen et al., 2023
In the citing paper
“Multiple iterative error correction is costly with limited benefits.(Chen et al., 2023)”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Chen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5 EXPERIMENTS
“When sampling multiple programs for a problem, we set temperature τ = 0.7, then we perform execution-based selection described in Section 2.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F CODE TRANSLATION F.1 BASELINE PROMPT (3-SHOT)
“The prompt for generating the initial Python translation is taken from (Chowdhery et al., 2022).We set temperature τ = 0.8 as in (Chowdhery et al., 2022).”
Citations
Cites
Marsden, W. (2012) · doi:10.1017/cbo9781139207249.009 · Cited by 170,911 · cited 2×
“In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks [8, 13, 38, 67, 57].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“We compare SELF-DEBUGGING to Graphix-T5 [31], T5-3B + Syn data [66] and T5-3B + N-best Reranking [64], which are adapted from the T5 architecture [42] and specially trained for text-to-SQL generation.”
MethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 3×
“For example, chain-of-thought prompting asks the model to produce intermediate reasoning steps in natural language, which can be accomplished either with few-shot exemplars (Wei et al., 2022) or in a zero-shot manner (Kojima et al.”
BackgroundSelf-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 5×
“One popular technique is reinforcement learning from human feedback (RLHF) (Ziegler et al., 2019; Ouyang et al., 2022), and RLHF-trained models have demonstrated the ability to avoid harmful outputs when instructed to do so in the prompt (Ganguli et al., 2023).”
MethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 5×
“The prompt for generating the initial Python translation is taken from (Chowdhery et al., 2022).”
BackgroundMethodInfluentialSelf-citationSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 2×
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Research works printed but not linked: 2
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In Interna- tional Conference on Learning Representations, 2019.
- Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, and Jishen Zhao. Coda: An end-to-end neural program decompiler. In NeurIPS, 2019.
Cited by
Sorted by citations
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Li et al. [87] employs the LLM in finding the failure-inducing test cases.”
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Chen et al. [150] introduced SELF-DEBUGGING, which teaches an LLM to debug its predicted code via few-shot learning, SELF-DEBUGGING reports baseline accuracy improvements of up to 12%.”
MethodEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
“Self-Debugging [6] employs a debugging component that is used to improve existing implementations given feedback from a code execution environment.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256
“Self-debugging [Chen et al., 2023] teaches LLM to perform rubber duck debugging after code generation.”
Automatic Prompt Optimization with “Gradient Descent” and Beam Search
Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C., & Zeng, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.494 · Cited by 191
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151
Self-citationLogic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144
“Recent works (Chen et al., 2023; Madaan et al., 2023) have adopted similar ideas to improve code generation, by teaching LLMs to debug their predicted programs via fewshot demonstrations.”
BackgroundLLM for SoC Security: A Paradigm Shift
Saha, D., Tarek, S., Yahyaei, K., Saha, S. K., Zhou, J., Tehranipoor, M. M., & Farahmandi, F. (2024) · IEEE Access · doi:10.1109/access.2024.3427369 · Cited by 128
“This has led to the emergence of several prompting techniques [115], [116], [117], [118], [119], [120], [121], [122], [123], [124], [125], [126], [127], [128], [129].”
BackgroundMethodDesigning Heterogeneous LLM Agents for Financial Sentiment Analysis
Xing, F. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3688399 · Cited by 88
“LLM collaboration is also more investigated on many general natural language processing tasks including sentiment analysis, but their applicability on FSA lacks direct evidence.”
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86
“…et al., 2023; Qin et al., 2023), while also showcasing surprising abilities of language understanding (Wei et al., 2022a; Begus et al., 2023), generation (Pu and Demberg, 2023; Lin and Chen, 2023; Lyu et al., 2023a), and reasoning (Wei et al., 2022b; Kojima et al., 2022; Dasgupta et al., 2022).”
BackgroundWhen Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72
“…et al., 2023; Yao et al., 2022b; Parisi et al., 2022; Tang et al., 2023b), and websites (Shi et al., 2017; Nakano et al., 2021; Yao et al., 2022a; Zhou et al., 2023b; Gur et al., 2023; Deng et al., 2023) as well as general code execution (Yang et al., 2023; Le et al., 2022; Ni et al., 2023).”
BackgroundGeneralized Planning in PDDL Domains with Pretrained Large Language Models
Silver, T., Dan, S., Srinivas, K., Tenenbaum, J. B., Kaelbling, L. P., & Katz, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.30006 · Cited by 150
LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 100
“Chen et al. [63] propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.”
BackgroundOlausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57
“Second, Logic-LM employs a self-refinement strategy, which has shown promise across code generation and NLP tasks (Zhang et al., 2023b; Chen et al., 2023a; Peng et al., 2023; Madaan et al., 2023; Olausson et al., 2023) but which we do not consider here.”
MethodCRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
PURPLE: Making a Large Language Model a Better SQL Writer
Ren, T., Fan, Y., He, Z., Huang, R., Dai, J., Huang, C., Jing, Y., Zhang, K., Yang, Y., & Wang, X. S. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00009 · Cited by 78
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 127
“Prior work shows that LLMs can improve their responses via feedback (Shinn et al., 2023; Paul et al., 2023; Madaan et al., 2023; Ganguli et al., 2023; Ma et al., 2023; Chen et al., 2023b; Peng et al., 2023; Kim et al., 2023; Du et al., 2023; Liang et al., 2023; Chen et al., 2023a; Pan et al., 2023).”
BackgroundClover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 94
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
Self-citation(2023) · doi:10.18653/v1/2023.emnlp-main.364 · Cited by 38
CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
Xu, R., Cao, J., Lu, Y., Wen, M., Lin, H., Han, X., He, B., Cheung, S.-C., & Sun, L. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1158 · Cited by 35
Applying fine-tuned LLMs for reducing data needs in load profile analysis
Hu, Y., Kim, H., Ye, K., & Lü, N. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124666 · Cited by 34
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.337 · Cited by 31
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“The environment can help unlock the capabilities like self-debugging [15] and self-reflection [53].”
BackgroundRL4F: Generating natural language feedback with reinforcement learning for repairing model outputs
Akyurek, A. F., Akyurek, E., Kalyan, A., Clark, P., Wijaya, D. T., & Tandon, N. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.427 · Cited by 29
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28
Self-citationJARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511593 · Cited by 36
An Approach for Rapid Source Code Development Based on ChatGPT and Prompt Engineering
Li, Y., Shi, J., & Zhang, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3385682 · Cited by 44
“This section incorporates the Self-Debugging method [32] for optimizing the Prompt in Fig.”
MethodYao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., & Yu, H. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.538 · Cited by 27
Chang, H., & Li, T. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101755 · Cited by 35
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29
“We evaluate F AC T OOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023; Chen et al., 2023).”
MethodLLMs cannot find reasoning errors, but can correct them given the error location
Tyen, G., Mansoor, H., Carbune, V., Chen, P., & Mak, T. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.826 · Cited by 156
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G. (2024) · arXiv · doi:10.48550/arxiv.2410.02089 · Cited by 23
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 61
“While early models 3,4 were characterized by their limited scope and functional capabilities, primarily tailored for specific tasks like text classification and sentiment analysis, the landscape of natural language processing (NLP) underwent a transformative shift with the introduction of neural…”
Large Language Model Operations (LLMOps): Definition, Challenges, and Lifecycle Management
Díaz-de-Arcaya, J., López-de-Armentia, J., Miñón, R., Ojanguren, I. L., & Torre-Bastida, A. I. (2024) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech61897.2024.10612341 · Cited by 22
“Model Selection [6], [15], [21], [22], [23], [27], [25], [28], [29] S3 Adaptation [6], [14], [15], [21], [22], [23], [24], [27], [25], [28], [30], [31] S4 Evaluation [14], [15], [22], [23], [24], [27], [25], [28], [32], [33], [34] S5 Deployment [15], [21], [22], [23], [27], [25], [28], [35] S6…”
BackgroundAutomating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
Alkafaween, U., Al-Bluwi, I., & Denny, P. C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13100 · Cited by 23
Tang, H., Key, D., & Ellis, K. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2243 · Cited by 20
Selective Demonstrations for Cross-domain Text-to-SQL
Chang, S., & Fosler‐Lussier, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.944 · Cited by 30
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 27
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
“…methods [196], [197], [198], [199], uncertainty estimation [147], [200], [201], [202], [203], [204], [205], controlled decoding [206], [207], [208], [209], self-refinement [210], [211], [212], [213], [214], and leveraging external knowledge during inference [175], [215], [216], [217], [218], [219].”
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“This approach is problematic as (1) closed-source APIs keep changing and have unpredictable availability (Pozzobon et al., 2023; Chen et al., 2023a), (2) it relies on the assumption that a more capable model exists (3) it can reinforce model hallucination (Gudibande et al., 2023) and (4) ,…”
BackgroundExploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Dai, Y. Q., Lang, H., Zeng, K., Huang, F., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.351 · Cited by 27
Iterative Prompt Refinement for Mining Gene Relationships from ChatGPT
Chen, Y., Gao, J., Petruc, M., Hammer, R., Popescu, M., & Xu, D. H. (2024) · International Journal of Artificial Intelligence and Robotics Research · doi:10.1142/s2972335324500054 · Cited by 15
Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap
Staniek, M., Schumann, R., Züfle, M., & Riezler, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00654 · Cited by 21
“With the advent of large language models and in-context learning, more recent work focuses on prompt engineering the task (Sun et al., 2023; Chen et al., 2023; Pourreza and Rafiei, 2023).”
BackgroundSänger, M., Mecquenem, N. D., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., & Kosch, T. (2024) · GigaScience · doi:10.1093/gigascience/giae030 · Cited by 20
COFFE: A Code Efficiency Benchmark for Code Generation
Peng, Y., Wan, J., Li, Y., & Ren, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715727 · Cited by 15
“Researchers have proposed different approaches to make use of LLMs on code generation via prompting engineering [10, 58, 70, 74, 94], multiple-agent cooperation [29, 30, 32, 80, 87, 92], and retrieval augmentation [50, 67, 77, 90, 95].”
Can ChatGPT support software verification?
Janßen, C., Richter, C., & Wehrheim, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3_13 · Cited by 19
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
“…coding capabilities of LLM without the need to adjust its parameters (Chen et al., 2022; Huang et al., 2023; Li et al., 2023a; Zhang et al., 2023c; Chen et al., 2023; Key et al., 2022) However, most of these works set up the evaluation in scenarios LLMs are familiar with, e.g., HumanEval (Chen et…”
Nguyen, M. T., Tran, K. T., Nguyen, N. V., & Vu, X.-S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.70 · Cited by 11
“Recent studies have explored various methods for fine-tuning LLMs with instruction data, enhancing their performance on specific tasks and domains (Koleva et al., 2022; Qiao et al., 2022; Li et al., 2023; Wu et al., 2023; Chen et al., 2023).”
Guo, C., Tian, Z., Tang, J., Li, S., & Wang, T. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103978 · Cited by 15
Luo, Q., Lin, Q., Xu, L., Wu, S., Mao, R., Wang, C., Feng, H., Huang, B., & Du, Z. (2026) · Big Earth Data · doi:10.1080/20964471.2026.2615511 · Cited by 11
Large Language Models Can Self-Correct with Key Condition Verification
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.714 · Cited by 36
“Other sources, such as compilers (Chen et al., 2024; Zeng et al., 2021) or search engines (Yu et al., 2023b) can provide external feedback.”
BackgroundMethodPlan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Liu, T., Guo, Q., Yang, Y., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.169 · Cited by 23
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
Chen, H., Saha, A., Hoi, S. C. H., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.417 · Cited by 18
Peng, Z., Kuai, X., Ke, S., Dong, X., & Guo, R. (2025) · Big Earth Data · doi:10.1080/20964471.2025.2483541 · Cited by 11
“Chen et al. (2023b) incorporated error messages as prompts for LLMs, enhancing accuracy through a self-debugging approach.”
ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8
“The effectiveness of causal reasoning pathways [11] is also crucial, ensuring that the output of LLMs is accurate but also transparent and verifiable.”
BackgroundLLM Critics Help Catch LLM Bugs
McAleese, N., Pokorny, R. M., Uribe, J. F. C., Nitishinskaya, E., Trebacz, M., & Leike, J. (2024) · arXiv · doi:10.48550/arxiv.2407.00215 · Cited by 10
“Improvements from self-correction and closely related methods have been claimed for harmlessness [1], factuality [24], computer control [14] and moral reasoning [5], as well as in other domains [20, 4, 7].”
BackgroundA Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8
Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.05080 · Cited by 7
MeshAgent: enabling reliable network management with large language models
Zhou, Y., Hsieh, K., Mani, S. K., Kandula, S., & Liu, Z. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771567 · Cited by 7
“Even advanced prompting fails on networking queries that require precise multi-step reasoning.”
Exploring Large Language Models in Information Systems: A Survey
Bianchini, F., & Marinacci, M. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_5 · Cited by 7
On the Challenges of Using Large Language Models for NCL Code Generation
Moraes, D. D. S., da Costa, P. B., Busson, A. J. G., Boaro, J. M. C., Neto, C. D. S. S., & Colcher, S. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia_estendido.2023.236175 · Cited by 9
“One of the main challenges is the creation or use of a sufficiently extensive dataset for training the LLM or fine-tuning a pre-trained model, as done in [8, 9, 14, 19, 22].”
BackgroundMethodTrojanSQL: SQL Injection against Natural Language Interface to Database
Zhang, J., Zhou, Y., Hui, B., Liu, Y., Li, Z., & Hu, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.264 · Cited by 8
“The emergence of powerful large language models (LLMs) has recently enabled the development of highly effective parsers with minimal demostra-tion examples (Chen et al., 2023), indicating the potential for LLM-based parsers to serve as novel interfaces for databases (Li et al., 2023).”
BackgroundMethodFully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 12
“Several studies explore an iterative approach to program synthesis in a manner similar to SEIDR [Chen et al., 2023; Shinn et al., 2023; Xia and Zhang, 2023].”
Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing
Yan, H., Srivastava, S., Tai, Y., Wang, S. I., Yih, W.-T., & Yao, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.177 · Cited by 15
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5
“The repair process can be aided with different artifacts, e.g., LLM-generated feedback on the code [27], error information produced by code execution [12, 14, 18].”
Fundamental Approaches to Software Engineering
Beyer, D., & Cavalcanti, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3 · Cited by 7
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1354 · Cited by 105
Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
Yang, Z., Zhang, Y., Wang, Y., Xu, Z., Lin, J., & Sui, Z. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.203 · Cited by 22
Flows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 34
Large Language Model Guided Self-Debugging Code GenerationPreprint
Adnan, M., Kuhn, C. C. N., & Xu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5396508 · Cited by 6
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Cheng, C.-A., Nie, A., & Swaminathan, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2287 · Cited by 4
The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models
Ruiz, F. V., Hort, M., & Moonen, L. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756966 · Cited by 13
“Successive feedback loops in the generation process have been shown to improve outcomes [3, 8].”
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-TalkPreprint
Ulmer, D., Mansimov, E., Lin, K., Sun, J., Gao, X., & Zhang, Y. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2401.05033 · Cited by 55
“…improvement in responses has been observed through the use of an external critique model (Paul et al., 2023), or through self-critiquing for a diverse set of tasks such as information extraction (Gero et al., 2023), code generation (Chen et al., 2023) and summarization (Saunders et al., 2022).”
MethodOn the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“These take many forms: common search algorithms with the LLM cast in the role of heuristic [37, 11], approaches which reduce error rates by enforcing various consistency conditions [8, 6, 17], and direct LLM self-critique [27, 35, 5, 15, 21, 38].”
MethodMind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3
Self-citationEnhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
Di, Y., & Zhang, T. (2025) · arXiv · doi:10.48550/arxiv.2505.07768 · Cited by 3
TraffiCoT-R: A framework for advanced spatio-temporal reasoning in large language models
Alsahfi, T., & Qasim, K. U. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.05.027 · Cited by 2
Exploring Large Language Models for Text-to-SQL Error Correction with LoRA Fine-Tuning
Jiang, G., & Ma, C. (2025) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace65325.2025.11018984 · Cited by 2
“Self-debugging [6] leverages the concept of rubber duck debugging [13] by providing additional column predictions and explanation for the original question and SQL to generate error feedback, guiding the model in error correction.”
SGEU: enhancing LLM reasoning via backward exemplar generation and verification
Wang, Z., Zhou, X., Yang, Y., Ma, B., Wang, L., & Dong, R. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06529-8 · Cited by 1
A Survey of Natural Language-Based Editing of Low-Code Applications Using Large Language Models
Gorissen, S. C., Sauer, S., & Beckmann, W. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64576-1_15 · Cited by 2
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3
“These works are often used to improve the capabilities of LLM agents, which are typically evaluated on simulated (Ruan et al., 2023) and real-world (Zhou et al., 2023b) environments.”
BackgroundSelf-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2024) · arXiv · doi:10.48550/arxiv.2410.04444 · Cited by 2
“Some agents are de-LLM signed to iterate over a fixed routine consisting of a list of fixed modules, while some of the modules are capable of taking self-or environment feedback to refine their actions (Chen et al., 2023b; Qu et al., 2024a; Tang et al., 2025).”
Chain-of-Program Prompting with Open-Source Large Language Models for Text-to-SQL
Xu, B., Li, S., Wu, Y., Wei, S., Du, M., Wang, H., & Song, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650552 · Cited by 3
“Researchers continuously propose better context learning prompts to help models parse questions more effectively [6]–[8], [15], [16], leading to superior performance.”
BackgroundMethodRunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 33
Metadata
- Authors
- Chen, X., Lin, M., Schärli, N., & Zhou, D.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.05128
- OpenAlex
- W4365205411
- Cited by
- 812
- References
- 75
- Citations by year
- 2026: 287 · 2025: 222 · 2024: 211 · 2023: 92
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, program repair, code explanation, text-to-SQL, sample efficiency, unit testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
