The DOI matches its DataCite record; 77 of the 77 references it cites were found in a registry or an index; 6 of the 16 sources its claims cite were checked for support.
Teaching Large Language Models to Self-Debug
Open accessAbstract
Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
SELF-DEBUGGING teaches LLMs to debug their own generated code via few-shot prompting, using execution results and natural-language code explanations without additional training. It improves state-of-the-art accuracy on text-to-SQL, code translation, and text-to-Python benchmarks, and notably improves sample efficiency.
Key findings
- On Spider text-to-SQL, where no unit tests are available, SELF-DEBUGGING with code explanation improves the Codex baseline from 81.3 to 84.1, and improves accuracy on the hardest problems by 9%.
- On TransCoder C++-to-Python and MBPP text-to-Python, using unit tests plus code explanation improves baseline accuracy by up to 12%; e.g., Codex on TransCoder improves from 80.4 to 92.5 with UT + Expl.
- SELF-DEBUGGING improves sample efficiency: on Spider, applying it to greedy decoding matches the baseline using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline using 32 samples.
- Error analysis shows Spider fixes often address wrong WHERE conditions (25.7%), missing DISTINCT (17.1%), and wrong JOIN clauses (14.3%); on TransCoder and MBPP, 60%–70% of successful fixes address output mismatch errors.
Methods
- The method uses few-shot prompting with a pretrained LLM, without finetuning, in iterative turns of Generation, Explanation, and Feedback; feedback can be simple, unit-test-based, code-explanation-based, or execution-trace-based.
- For multiple initial samples, the paper selects the program with the most frequent execution result among those without execution errors, filtering by unit tests when available; maximum debugging turns is 10, though successful debugging mostly ends within 3 turns.
- Evaluation uses Codex, GPT-3.5, GPT-4, and StarCoder on Spider development set, TransCoder C++-to-Python with 560 problems and 10 unit tests each, and MBPP with 500 Python problems and 3 unit tests each, with the first MBPP unit test included in the prompt.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: on MBPP, context length limits reduce the number of exemplars when adding code explanation (6 exemplars for simple/unit-test feedback vs. 3 for unit test + explanation), and code explanation is less significant when the initial predicted code is far from correct.
- Evident from the text: GPT-4 is tuned for zero-shot prompting and does not follow few-shot exemplars as well as Codex on Spider; evaluation is limited to specific benchmarks and C++-to-Python translation rather than broader languages or tasks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
95 Own result · 19 Own interpretation · 9 No basis given · 4 Partly supported · 4 Sources not found · 2 Own method · 1 Supported · 1 Supported so far
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“On the other hand, large language models have been shown to be capable of describing their generated problem solutions in both text (Wei et al., 2022;Kojima et al., 2022;Zhou et al., 2023) and code (Gao et al., 2022;Chen et al., 2022) formats.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Code Explanation feedback (Expl).
What the sources say
- Not checked — no full text of the sourcecites Wei et al., 2022 source
- Supportedcites Gao et al., 2022 source
The Gao et al. citation supports the code-format part of the sentence. The Introduction (Section 1) says PAL uses an LLM to generate programs as reasoning steps for natural-language problems; the Python interpreter then executes those steps to obtain the final answer. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that SELF-DEBUGGING consistently improves the performance.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.1 MAIN RESULTS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As will be shown in Section 5, it is insufficient for accurately predicting SQL correctness when the few-shot prompt only contains simple feedback.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 4.1 TEXT-TO-SQL GENERATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, on Spider, applying SELF-DEBUGGING to predictions generated with greedy decoding matches the baseline accuracy using 16 samples, and SELF-DEBUGGING from 8 samples outperforms the baseline accuracy using 32 samples.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.1 SELF-DEBUGGING IMPROVES THE SAMPLE EFFICIENCY
plainn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the Spider benchmark where there are no unit tests to verify the correctness of predictions, SELF-DEBUGGING with code explanation consistently improves the baseline by 2 -3%”
- Also stated in: § 1 INTRODUCTION · plain
“On the Spider benchmark (Yu et al., 2018) for text-to-SQL generation where there are no unit tests in the problem description, with code explanation, SELF-DEBUGGING consistently improves the baseline by 2 -3% with different numbers of initial programs”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, we observe that the initial SQL queries generated by LLMs are usually not completely wrong, but they tend to make small mistakes when the questions require more complex SQL queries, e.g., missing a few WHERE conditions or SELECT columns.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, on extra hard problems, SELF-DEBUGGING increases the accuracy by 9%.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Without unit test execution, both models tend to be overconfident in their own initial predictions, while GPT-4 is better than GPT-3.5 on Python generation.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.2 IMPORTANCE OF CODE EXECUTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to Codex, few-shot prompting does not notably improve the SELF-DEBUGGING performance of GPT-3.5 and GPT-4, and thus their SELF-DEBUGGING procedures fully rely on their internal code knowledge.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.2 IMPORTANCE OF CODE EXECUTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Section 5.2.2, we further present an ablation study showing that leveraging code execution is crucial for enabling a performance leap with SELF-DEBUGGING.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 27
- The quote only partly states this claim. 9
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Different from prior works on utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al., 2023a;Austin et al., 2021), SELF-DEBUGGING teaches the model to identify the implementation errors via investigating into execution results and explaining the code by itself.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 1 INTRODUCTION
What the sources say
- Not matchedcites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Austin et al., 2021 source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior works demonstrate that decoding multiple samples can significantly improve the performance of large language models (Wang et al., 2023;Shi et al., 2022).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution-based code selection.
What the sources say
- Not matchedcites Wang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Shi et al., 2022 source
Section 4.3 (Primary Results; page unavailable) states that more samples improve performance for most selection methods and that MBR-EXEC significantly outperforms baselines without selection. The paper notes an exception: MALL performs worse with larger sample sizes. That exception is consistent with the claim that multiple samples *can* improve performance. (LLM, conf=0.91)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In particular, for code generation tasks, we can utilize code execution to select the final prediction (Chen et al., 2019;Li et al., 2022;Shi et al., 2022;Zhang et al., 2022;Ni et al., 2023).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution-based code selection.
What the sources say
- Not matchedcites Chen et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Li et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Shi et al., 2022 source
The Introduction describes sampling candidate programs and selecting a single prediction based on their execution results. Section 3 likewise defines the proposed approach as selecting the best candidate from code-model samples, directly supporting the claim. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work on code repair has demonstrated that training the repair model on execution traces improves the debugging performance (Wang et al., 2018;Gupta et al., 2020).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution trace feedback (Trace).
What the sources say
- Not matchedcites Wang et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Gupta et al., 2020 source
In Results (p. 6), the authors report that the debugger with TraceEmbed had the lowest repair error at every tested number of mutations, demonstrating that execution traces were helpful. The mutation-benchmark discussion (p. 7) also reports mostly better performance with trace embedding. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 2 PROMPTING FOR CODE GENERATION
What the sources say
- Supportedcites Brown et al., 2020 source
Section 2, Approach (p. 6) defines few-shot learning as giving the model several task demonstrations at inference time. It specifies that each demonstration pairs a context with a desired completion, after which the model is expected to complete a new context. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“and does not produce meaningful changes to the initial predicted SQL queries.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that on TransCoder and MBPP benchmarks where at least one unit test is available for SELF-DEBUGGING, simple feedback also utilizes the execution result to infer the code correctness even if the execution information is not presented in the feedback message, thus simple feedback still improves the model prediction for both applications.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Although GPT-4 is shown to be much stronger than previous GPT models on many tasks (Ope-nAI, 2023), on Spider, both its initial SQL generation and SELF-DEBUGGING performance are much worse than Codex.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite that the baseline performance of StarCoder is considerably worse than GPT models, SELF-DEBUGGING with unit test execution also offers a significant performance gain, i.e., 6% on MBPP with the execution trace feedback.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all tasks, models generally benefit from richer feedback for SELF-DEBUGGING, especially when the execution information is present in the feedback.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meanwhile, by leveraging feedback messages and reusing failed predictions, SELF-DEBUGGING notably improves sample efficiency, and can match or outperform baseline models that generate more than 10× candidate programs.”
- Also stated in: § 1 INTRODUCTION · plain
“Meanwhile, SELF-DEBUGGING improves sample efficiency, and can match or outperform baseline models that sample more than 10× predictions.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though empirically the successful debugging processes mostly end within 3 turns”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5 EXPERIMENTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar sample efficiency improvement on other benchmarks, and we defer the discussion to Appendix C.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.1 SELF-DEBUGGING IMPROVES THE SAMPLE EFFICIENCY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that typically one debugging turn is sufficient, and the accuracy improvement after one turn is within 0.1%.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5.2.1 SELF-DEBUGGING IMPROVES THE SAMPLE EFFICIENCY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments, we will demonstrate that leveraging unit tests whenever available consistently boosts the SELF-DEBUGGING performance.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution-based code selection.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meanwhile, its self-debugging performance gain is on par with Codex: over 12% improvement on Transcoder, and 8% improvement on MBPP.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > return cnt
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, we demonstrate that SELF-DEBUGGING can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong COUNT columns | 5.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 9: An example where SELF-DEBUGGING with unit test feedback fixes the code translation error, while the simple feedback fails.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > C.2 TEXT-TO-PYTHON GENERATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the translation with code explanation correctly checks for the end of a string.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > + Expl.
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing nested conditions | 5.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong WHERE conditions | 25.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong INTERSECT/UNION clauses | 8.6”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Output mismatch Runtime errors | 61.9 38.1 | 69.2 30.8”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong column selection | 5.7”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“See Figure 11 in Appendix D for an example where SELF-DEBUGGING makes major changes to the initial code.”
- Also stated in: § Original Python code · plain
“Figure 11: An example on MBPP where the prediction after SELF-DEBUGGING is very different from the initial code.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > C.2 TEXT-TO-PYTHON GENERATION
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Original Python code · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 10: An example where adding code explanation results in the correct code translation, while the baseline prompt does not.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > + Expl.
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Wrong aggregate functions and keywords 5.8”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also performs close to GPT-3.5 on MBPP, and outperforms it for both Simple and UT.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing the DISTINCT keyword Wrong JOIN clauses Wrong number of SELECT columns | 17.1 14.3 11.4”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F.5 UNIT TEST + EXPL. FEEDBACK PROMPT (2-SHOT)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“prior works suggest that such large language models are not yet capable of correcting code when lacking external feedback, such as unit tests or human instructions (Chen et al., 2023a).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 1 INTRODUCTION
What the sources say
- Not matchedcites Chen et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, text-to-SQL generation represents the scenario where no unit tests are available, where SELF-DEBUGGING enables the model to identify errors and justify code correctness by explaining the predicted code.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 4 APPLICATIONS In our evaluation, we showcase applications of SELF-DEBUGGING to different code generation domains: text-to-SQL generation, code translation, and text-to-Python generation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This debugging process is reminiscent of rubber duck debugging for human programmers, where explaining the code line-byline in natural language to a rubber duck significantly boosts debugging efficiency without expert guidance (Hunt & Thomas, 2000).”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 1 INTRODUCTION
What the sources say
- Not matchedcites Hunt & Thomas, 2000
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since the unit tests are not available, it becomes more challenging for the model to infer the correctness of the predicted SQL queries.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 4.1 TEXT-TO-SQL GENERATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that both the execution trace and the line-by-line explanation come from model generation instead of code execution, thus the trace feedback does not require more information than the pure code explanation feedback; i.e., no access to intermediate execution states.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Execution trace feedback (Trace).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, we demonstrate that SELF-DEBUGGING empowers the model to perform rubber duck debugging, so that the model can identify and fix bugs without human instructions.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(Note that after this kind of feedback, we finish SELF-DEBUGGING and do not continue with”
§ Write Python function to complete the task and pass the assertion tests.### Task Start ### # These are the assertions for your function: assert count_ways(2) == 3 """ Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x n board.""" def count_ways(n): > Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, SELF-DEBUGGING teaches the large language model to notice prediction errors via code explanation in the style of rubber duck debugging, which does not require the feedback message to explicitly explain the implementation errors and how to fix them.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The new code prediction is used only if it also passes the unit test, otherwise the original prediction which passes will be used for evaluation.”
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
“The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.”
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
“The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.”
§ Write Python function to complete the task and pass the assertion tests.### Task Start ### # These are the assertions for your function: assert count_ways(2) == 3 """ Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x n board.""" def count_ways(n): > Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § Task End ### ### Task Start ### # These are the assertions for your function: assert square_nums( [1,2,3,4,5,6,7,8,9, 10])== [1,4,9,16,25,36,49,64,81,100] """ Write a function to find squares of individual elements in a list using lambda function.""" def square_nums(nums): square_nums = list(map(lambda x: x ** 2, nums)) return square_nums · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our work highlights the promise of improving the coding performance of large language models by teaching them to iteratively debug their own predictions, instead of requiring the model to generate the correct code from scratch.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we focus on code generation applications, and SELF-DEBUGGING demonstrates the effectiveness of both self-generated feedback and unit test feedback acquired by code execution.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, SELF-DEBUGGING enables the model to generate feedback messages on its own at test time, and does not require extra training.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > Self-debugging with explanation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper relies on this without support.
For complex programming tasks, generating the correct solution in one go is challenging.
“However, for complex programming tasks, generating the correct solution in one go becomes challenging”
“However, for many programming tasks, generating correct code with a single attempt is challenging.”
The paper relies on this without support.
Human programmers typically look into incorrect code and investigate execution results, then make changes to resolve implementation errors.
“Instead of completely discarding the incorrect code, humans typically look into the code and investigate execution results, then make changes to resolve the implementation errors.”
The paper relies on this without support.
There is no guarantee that code written on the first try by human programmers is always accurate.
“Intuitively, even for human programmers, there is no guarantee that the code written on the first try is always accurate.”
The paper relies on this without support.
Human programmers can usually identify bugs by describing code implementation and comparing it to the problem description without extra guidance.
“By describing the code implementation and comparing it to the problem description, human programmers are usually able to identify the bugs without extra guidance.”
The paper relies on this without support.
The Spider benchmark's difficulty annotations, based on the complexity of the ground truth SQL queries, are valid.
“where the difficulty of each problem is annotated in the Spider benchmark based on the complexity of the ground truth SQL queries.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“AlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [7]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We evaluate FACTOOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023;Chen et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous works (Chen et al., 2023;Zhang et al., 2024a) rely on generating test cases to evaluate code quality.However, this method has issues with inaccurate test case generation (Jain et al., 2024), inability to handle special judge (Quan et al., 2025), and failure to evaluate non-code content in response such as code comments and explanations.”
From RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation (2025) · cites this paper as Chen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Meanwhile, recent research shows that LLMs such as ChatGPT demonstrate preliminary code debugging capabilities (Chen et al., 2023b,a).”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Chen et al., 2023b
The cited source says what the claim attributes to it.
In the citing paper
“Since ChatGPT has shown preliminary capability in error detection and correction during code generation (Shinn et al., 2023;Chen et al., 2023b;Kim et al., 2023;Nair et al., 2023;Madaan et al., 2023), we use Chat-GPT to perform multiple rounds of self-debugging and investigate the impact on DSR.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Chen et al., 2023b
The cited source says what the claim attributes to it.
In the citing paper
“In terms of optimization, the SELF-DEBUGGING approach [14] leads the post-code generation phase.It endows LLMs with the ability to debug their output, reinforcing the theme of continual refinement in the generated code.”
From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023;Chen et al., 2023c;Lu et al., 2023;Kim et al., 2023).”
From Large language models as tool makers (2023) · cites this paper as Chen et al., 2023c
The cited source says what the claim attributes to it.
In the citing paper
“Various refinement strategies use external feedback from knowledge bases (Peng et al., 2023;Chern et al., 2023), code interpreters (Chen et al., 2023;Zhang et al., 2023), tool outputs (Gou et al., 2024;Schick et al., 2023), or separately trained reward models (Akyürek et al., 2023).”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Chen et al., 2023
In the citing paper
“Multiple iterative error correction is costly with limited benefits.(Chen et al., 2023)”
From CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025) · cites this paper as Chen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > 5 EXPERIMENTS
“When sampling multiple programs for a problem, we set temperature τ = 0.7, then we perform execution-based selection described in Section 2.”
§ TEACHING LARGE LANGUAGE MODELS TO SELF-DEBUG > F CODE TRANSLATION F.1 BASELINE PROMPT (3-SHOT)
“The prompt for generating the initial Python translation is taken from (Chowdhery et al., 2022).We set temperature τ = 0.8 as in (Chowdhery et al., 2022).”
Citations
Cites
Marsden, W. (2012) · doi:10.1017/cbo9781139207249.009 · Cited by 170,911 · cited 2×
“In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks [8, 13, 38, 67, 57].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“We compare SELF-DEBUGGING to Graphix-T5 [31], T5-3B + Syn data [66] and T5-3B + N-best Reranking [64], which are adapted from the T5 architecture [42] and specially trained for text-to-SQL generation.”
MethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 3×
“For example, chain-of-thought prompting asks the model to produce intermediate reasoning steps in natural language, which can be accomplished either with few-shot exemplars (Wei et al., 2022) or in a zero-shot manner (Kojima et al.”
BackgroundSelf-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 5×
“One popular technique is reinforcement learning from human feedback (RLHF) (Ziegler et al., 2019; Ouyang et al., 2022), and RLHF-trained models have demonstrated the ability to avoid harmful outputs when instructed to do so in the prompt (Ganguli et al., 2023).”
MethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 5×
“The prompt for generating the initial Python translation is taken from (Chowdhery et al., 2022).”
BackgroundMethodInfluentialSelf-citationSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 2×
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Research works printed but not linked: 2
- Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In Interna- tional Conference on Learning Representations, 2019.
- Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, and Jishen Zhao. Coda: An end-to-end neural program decompiler. In NeurIPS, 2019.
Cited by
Sorted by citations
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Li et al. [87] employs the LLM in finding the failure-inducing test cases.”
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Chen et al. [150] introduced SELF-DEBUGGING, which teaches an LLM to debug its predicted code via few-shot learning, SELF-DEBUGGING reports baseline accuracy improvements of up to 12%.”
MethodEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315
Metadata
- Authors
- Chen, X., Lin, M., Schärli, N., & Zhou, D.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.05128
- OpenAlex
- W4365205411
- Cited by
- 812
- References
- 75
- Citations by year
- 2026: 287 · 2025: 222 · 2024: 211 · 2023: 92
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, program repair, code explanation, text-to-SQL, sample efficiency, unit testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
