The DOI matches its DataCite record; 47 of the 53 references it cites were found in a registry or an index; 12 of the 22 sources its claims cite were checked for support.
Self-Refine: Iterative Refinement with Self-Feedback
Open accessAbstract
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
SELF-REFINE improves large language model outputs by iteratively generating self-feedback and refining the same model’s output without supervised training, additional training, or reinforcement learning. Across 7 diverse tasks, it improves over one-step generation by ∼20% absolute on average, with especially large gains on preference-based tasks.
Key findings
- SELF-REFINE consistently improved GPT-3.5, ChatGPT, and GPT-4 across all evaluated tasks. Examples include GPT-4 Dialogue Response Generation improving from 25.4 to 74.6 (+49.2), GPT-4 Code Optimization from 27.3 to 36.0 (+8.7), and GPT-4 Constrained Generation from 15.0 to 45.0 (+30.0).
- Feedback quality matters: in Code Optimization, SELF-REFINE feedback scored 27.5 vs. 26.0 for generic feedback and 24.8 for no feedback; in Sentiment Reversal, 43.2 vs. 31.2 vs. 0; in Acronym Generation, 56.4 vs. 54.0 vs. 48.0.
- Iterative refinement generally improves outputs, with diminishing returns: Code Optimization rose from 22.0 at y0 to 28.8 at y3; Sentiment Reversal from 33.9 to 36.8; Constrained Generation from 29.0 to 49.7.
- Math Reasoning gains were modest: GPT-3.5 64.1 to 64.1 (0), ChatGPT 74.8 to 75.0 (+0.2), GPT-4 92.9 to 93.1 (+0.2). With oracle feedback, GPT-3 improved by 4.8% and GPT-4 by 0.7%.
Methods
- SELF-REFINE alternates between FEEDBACK and REFINE using the same model M: initial output y0 = M(pgen ∥ x); feedback fbt = M(pfb ∥ x ∥ yt); refinement yt+1 = M(prefine ∥ x ∥ y0 ∥ fb0 ∥ … ∥ yt ∥ fbt). It iterates until a stopping condition or up to a maximum of 4 iterations, using few-shot prompts and no supervised training, additional training, or reinforcement learning.
- Evaluation covered 7 tasks: Dialogue Response Generation, Code Optimization, Code Readability Improvement, Math Reasoning, Sentiment Reversal, Acronym Generation, and Constrained Generation (CommonGen-Hard). Base LLMs were GPT-3.5 (text-davinci-003), ChatGPT (gpt-3.5-turbo), GPT-4, and CODEX for code tasks. Metrics included task-specific metrics, human preference, and GPT-4 preference.
- Analysis included ablations on feedback quality and iteration count, comparison against generating multiple outputs, testing Vicuna-13B, and a qualitative error analysis of 70 samples for Code Optimization and Math Reasoning.
Limitations
- Stated: the base models need sufficient few-shot modeling or instruction-following abilities to provide feedback and refine in-context; Vicuna-13B struggled to generate feedback in the required format and often failed to refine properly.
- Stated: experiments used non-open-sourced models (GPT-3.5, ChatGPT, GPT-4, CODEX), whose pretraining corpus, sizes, and biases are not fully described, and which are not free to use; datasets were exclusively English; the approach does not explicitly guard against bad actors steering models toward toxic or harmful text.
- Evident from the text: human A/B evaluations were conducted by the authors, and GPT-4 was used as a proxy for human preference with reported correlations of 82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Generation. The paper also notes Math Reasoning gains are limited by the model’s inability to accurately identify errors, with ChatGPT feedback being “everything looks good” for 94% of instances.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
70 Own result · 25 Own interpretation · 9 No basis given · 7 Partly supported · 5 Own method · 4 Supported · 4 Sources not found · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“In code-generation tasks, SELF-REFINE improves the initial generation by up to absolute 13% when applied to strong code models such as Codex (code-davinci-002; Chen et al., 2021).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Refine Feedback
What the sources say
- Supportedcites Chen et al., 2021 source
The Chen et al. citation sits in the parenthetical identifying Codex, rather than after the reported SELF-REFINE improvement. Section 3, “Code Fine-Tuning” (p. 4), describes producing Codex and evaluating it on code-generation tasks, supporting that identification. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, these reinforcement learning methods require updating the model's parameters, unlike SELF-REFINE.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Supportedcites Stiennon et al., 2020 source
The Introduction (p. 2) says the policy is updated using PPO, and §3.4 (p. 6) describes a learned RL policy with parameters. This supports the claim about the reinforcement-learning methods discussed; the contrast with SELF-REFINE is the citing authors’ framing, not a finding attributed to this source. (LLM, conf=0.90)
- Not matchedcites Lu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Stiennon et al. (2020) source
The Introduction (p. 2) says the policy is trained through reinforcement learning and updated using PPO. Section 3.4 (p. 6) also describes the learned RL policy as having parameters. The comparison with SELF-REFINE is the citing authors’ own framing, not a claim this source is needed to establish. (LLM, conf=0.92)
- Not matchedcites Lu et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al. (2022a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This adjustment notably enhanced performance in the Math Reasoning task, with GPT-3 improving by 4.8% and GPT-4 by 0.7% Table 9.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
What the sources say
- Partly supportedcites Welleck et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.5 (pp. 8–9) reports that correcting only incorrect outputs and adding explicit feedback improve results in math program synthesis. It does not report the claimed 4.8% GPT-3 or 0.7% GPT-4 gains; its math results use different models and figures. The cited paper’s Table 9 concerns E2E hyperparameters, not those gains. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These methods differ from SELF-REFINE (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Partly supportedcites Stiennon et al., 2020 sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says this paper’s RL policy is updated using a reward for the entire generated summary, and a note on p. 6 explicitly says the reward model gives no rewards at intermediate time steps. That supports the distinction for this method, but not the broader claim that reinforcement-learning approaches generally cannot access feedback on intermediate generations. (LLM, conf=0.84)
- Not matchedcites Lu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked to generate a sentence containing up to 30 given concepts.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 4 demonstrates that on average, the quality of the output improves as the number of iterations increases.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all evaluated tasks, outputs generated with SELF-REFINE are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ∼20% absolute on average in task performance.”
- Also stated in: § 3.3 Results · plain
“SELF-REFINE consistently improves over base models across all model sizes, and additionally outperforms the previous state-of-the-art across all tasks.”
- Also stated in: § Refine Feedback · plain
“We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo;OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.3 Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, SELF-REFINE leads to especially high gains.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in the Code Optimization task, the initial output (y 0 ) has a score of 22.0, which improves to 28.8 after three iterations (y 3 ).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 35
- The quote only partly states this claim. 5
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“CommonGen-Hard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only the 3-5 related concepts given in CommonGen.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > K New Tasks
What the sources say
- Partly supportedcites Lin et al., 2020 sourceExperimental — accuracy not yet measured
Section 3.1 says CommonGen uses concept sets of three to five concepts, and Section 2 says the output should be a grammatical sentence using all given concepts. Our copy does not mention CommonGen-Hard or a requirement to incorporate 20–30 concepts (searched for “CommonGen-Hard” and “20-30 concepts”), so that part is unsupported. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing literature (Ouyang et al., 2022) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
What the sources say
- Partly supportedcites Ouyang et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.5 (page unavailable) calls the pretrained models’ behavior “poorly characterized,” which lends some support to the concern about incomplete characterization. But the paper does not say that existing literature omits the listed details; its Introduction gives the model sizes, and Section 5.3 discusses biased outputs. The claimed lack of disclosure about model sizes and biases is therefore not supported. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirically, we evaluated SELF-REFINE using the same base model of GPT-3 as Self-Correction, and with the same settings on the GSM8K benchmark.Self-Correction achieved 45.9% accuracy while SELF-REFINE (this work) achieved 55.7% (↑9.8).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Partly supportedcites Welleck et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) reports 45.92% accuracy on GSM with a GPT-3 Instruct generator and self-corrector, supporting the rounded 45.9% figure. The paper does not report SELF-REFINE’s 55.7% result or establish that the two methods used the same settings, so it does not substantiate the 9.8-point comparison. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Self-Correction trains a separate refiner (or”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Supportedcites Welleck et al., 2022 source
Section 2 (p. 2) says Self-Correction keeps the generator separate and trains the corrector with different objectives for different task requirements. The Introduction (pp. 1–2) also describes a trained corrector and its use across three tasks. The comparison describing SELF-REFINE’s prompting approach is not a claim about this source. (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Few-shot prompting (also referred to as”
What the sources say
- Supportedcites Brown et al., 2020 source
Section 2 Approach (p. 6) defines few-shot as giving the model K task examples as conditioning. It describes each example as a context and desired completion—the input-output pairs in the claim—followed by a final context for the model to complete. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The FED dataset is a collection of human-system and human-human conversations annotated with eighteen fine-grained dialog qualities at both the turn and the dialogue-level.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
What the sources say
- Supportedcites Mehri and Eskenazi, 2020 source
The Introduction describes the FED dataset as human quality annotations of human-system and human-human conversations. Section 3 (Data Collection) specifies eighteen fine-grained qualities: eight annotated at the turn level and ten at the dialogue level. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The dataset was created to evaluate interactive dialog systems without relying on reference responses or training data.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
What the sources say
- Partly supportedcites Mehri and Eskenazi, 2020 sourceExperimental — accuracy not yet measured
The Introduction says the FED dataset was collected for fine-grained evaluation of interactive dialog and contains no training data. It also says the dataset is intended to benchmark automatic metrics against human judgments; the absence of reference responses is a property of the FED metric, not a stated purpose or property of the dataset. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This effect is more pronounced in tasks like Sentiment Transfer, where changing from our feedback to generic feedback leads to a significant performance drop (43.2 to 31.2), and the task fails without feedback.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Code Optimization, performance slightly dips from 27.5 (SELF-REFINE feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINEand ChatGPT+SELF-REFINE across all tasks, even in tasks where the initial base results of GPT-4 were lower than GPT-3.5 or ChatGPT.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To address this, SELF-REFINE's feedback generates explicit numerical scores to capture the different aspects of output quality.This allows for a more balanced evaluation of outputs and the selection of the most appropriate one.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Vicuna-13b was able to consistently follow the task initialization prompt.However, it struggled to follow the prompts intended for feedback and refinement.This often led to outputs that resembled assistant-like responses, a representative example of which can be found in Appendix G.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > G Evaluation of Vicuna-13b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 12 breaks down errors found in the refinement stage of SELF-REFINE.Errors include: not being robust (10%), ignoring feedback (25%), and introducing a new problem (20%).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback and Refinement Error Analysis for Response Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that nearly all of GPT-4 gains are statistically significant, ChatGPT gains are significant for 4 out of 7 datasets, and GPT-3.5 gains are significant for 3 out of 7 datasets.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: GPT-4 is capable of assessing and providing reasoning for complex tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE demonstrates its iterative feedback and refinement capabilities in the context of website layout generation.ChatGPT initially produces a rudimentary layout for a given topic, and then uses the FEEDBACK to suggest specific, actionable improvements, as demonstrated in Figures 7 and 9.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > I Beyond Benchmarks
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Starting from a rudimentary initial design, SELF-REFINE refines HTML, CSS, and JS to evolve the website in terms of both usability and aesthetics.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Going Beyond Benchmarks
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the Dialogue Response Generation task, SELF-REFINE can ignore good feedback but in a majority of cases, it is robust to bad feedback and ignores bad feedback.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, SELF-REFINE achieves superior performance while using only 4 samples at most, significantly fewer than the 16 and 32 samples employed by other models.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > F Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance may not always monotonically increase with iterations: in multi-aspect feedback tasks like Acronym Generation, where the output quality can vary during iteration with improvement in one aspect but decline in another aspect.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 11 reports the occurrence of different types of errors in our sample, which includes Incorrect Feedback (25%), Generic Feedback (30%), and Incorrect Scoring (10%).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback and Refinement Error Analysis for Response Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A similar selection is possible for other tasks like Math Reasoning and Sentiment Reversal, while we observe that output quality increases monotonically with iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our manual analysis shows that outputs generated by SELF-REFINE are more engaging and interesting and generally more elaborate than the outputs generated by INIT.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE w/ GPT-3 | 55.7”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CommonGen-Hard requires models to generate coherent and grammatically correct sentences incorporating 20-30 concepts, as opposed to the original task which presents a set of 3-5 related concepts.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE wins | 36.0 | 48.0 | 54.0”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“INIT wins | 23.0 | 18.0 | 16.0”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE w/ GPT-4 | 94.5”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“PaL w/ GPT-4 | 93.3”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE -feedback | 1 | 10.1 | 62.15 | 3.03”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“SELF-REFINE | 1 | 15.3 | 59.64 | 2.90”
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the FEEDBACK step of SELF-REFINE.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For dramatic response evaluation, we found that the preference rate went down drastically to 58.92%, from 80.09%.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > O Math Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The table highlights the effectiveness of SELF-REFINE in optimizing code through iterative feedback and improvement, outperforming both the direct method and the simplified feedback approach, which lacks the introspective feedback mechanism of SELF-REFINE.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Direct | - | 9.7 | 62.29 | 3.09”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“SELF-REFINE | 2 | 15.6 | 65.60 | 3.74”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“PaL w/ ChatGPT | 74.2”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sentiment Transfer | 75.00 | 21.43 | 3.57”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“Acronym Generation | 44.59 | 12.16 | 43.24”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“Response Generation | 47.58 | 19.66 | 32.76”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Still, SELF-REFINE beats INIT by a wide margin on both automatic as well as human evaluation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At T = 0.0, SELF-REFINE produces less meaningful variables, less comments per line but even more function units.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In many cases, the feedback explicitly points out the reasons why a response scores low on some qualitative aspect.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At T = 0.7, SELF-REFINE produces more meaning variables, more function units and slightly more comments compared to the human annotators on average.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, this task has an extremely large number of reasonable outputs, and thus SELF-REFINE allows to better explore the space of possible outputs.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe that this task benefits significantly from SELF-REFINE because there are more opportunities to miss some of the concepts on the first attempt, and thus SELF-REFINE allows the model to fix these mistakes subsequently.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, the experiments in this work were performed with language models that are not open-sourced, namely GPT-3.5, ChatGPT, GPT-4, and CODEX.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Another limitation of our work is that we exclusively experiment with datasets in English.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE focuses on iterative creation with introspective feedback, making it suitable for evaluating the effectiveness of language models on the CommonGen-Hard task.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > K New Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This demonstrates the potential of SELF-REFINE in real-world, complex, and creative tasks.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Going Beyond Benchmarks
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Compared to a concurrent work, Reflexion (Shinn et al., 2023), our approach involves correction using feedback, whereas their setup involves finding the next best solution in planning using ReAct.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Not matchedcites Shinn et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the limited performance of Vicuna-13b suggests that this model may require more extensive prompt-engineering for optimal performance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > G Evaluation of Vicuna-13b
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, we release our code and model outputs to ensure the reproducibility of our work.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other languages, the current models may not provide the same benefits.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, having multiple FEEDBACK-REFINE iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This distinction allows our method to offer more precise and actionable feedback, making it suitable for a wider range of natural language generation tasks, including those that may not necessarily involve step-by-step planning such as open-ended dialogue generation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We thus hypothesize that since Vicuna-13B was trained on conversations, it does not generalize as well as instruction-based models to test-time few-shot tasks.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
What the sources say
- Not matchedcites Chiang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirectSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The setup was blind, i.e., the judges did not know which outputs were generated by which method.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > C Human Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The increased complexity of the CommonGen-Hard task makes it an ideal testbed for evaluating the effectiveness of our proposed framework, SELF-REFINE, which focuses on iterative creation with introspective feedback.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The judges were not aware of the method that generated each sample.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, acronym generation is a natural method testbed for our approach.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Acronym generation requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.
- Rests on an assumption: Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.
In the paper
“The primary objective of PIE is to optimize a given program by implementing algorithmic modifications that lead to improved runtime performance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
What the sources say
- Not matchedcites Madaan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through iterative creation and self-reflection, the SELF-REFINE framework empowers language models to progressively enhance the quality of their output, closely mimicking the human creative process and demonstrating its ability to improve generated text on complex and demanding natural language generation tasks like CommonGen-Hard (Figure 15).”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility.
In the paper
“Performance-Improving Code Edits or PIE (Madaan et al., 2023) focuses on enhancing the efficiency of functionally correct programs.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
What the sources say
- Not matchedcites Madaan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This significant increase in the number of concepts tests the model's ability to perform advanced commonsense reasoning, contextual understanding, and creative problem-solving, as it must generate meaningful sentences that encompass a broader range of ideas.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
Large language models often fall short in addressing intricate requirements.
“Although large language models (LLMs) can generate coherent outputs, they often fall short in addressing intricate requirements.”
The paper relies on this without support.
Math errors can be nuanced and sometimes limited to a single line or incorrect operation.
“In math, errors can be nuanced and sometimes limited to a single line or incorrect operation.”
The paper relies on this without support.
GPT-4 is capable of assessing and providing reasoning for complex tasks.
“In light of the impressive achievements of GPT-4 in assessing and providing reasoning for complex tasks, we leverage its abilities for evaluation in SELF-REFINE.”
The paper relies on this without support.
Code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase.
“though not related to the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase.”
The paper relies on this without support.
Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance.
“Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach.”
The paper relies on this without support.
Automated metrics may not provide an accurate assessment of text generation tasks.
“However, we acknowledge that automated metrics may not provide an accurate assessment of text generation tasks and rely on human evaluation instead.”
The paper relies on this without support.
Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics.
“Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics.”
The paper relies on this without support.
text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts, making GPT-3.5 a strong baseline.
“text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts and hence GPT-3.5 is a strong baseline.”
The paper relies on this without support.
Acronym generation requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.
“Like in email writing, acronym generation also requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.”
The paper relies on this without support.
Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.
“Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.”
The paper relies on this without support.
Some acronym quality criteria are difficult to quantify and are a matter of human preference.
“Some of these criteria are difficult to quantify, and are a matter of human preference.”
The paper relies on this without support.
Modern LLMs have superior instruction following capabilities.
“As with other modules, we leverage the superior instruction following capabilities of modern LLMs to instead provide a few demonstrations of each task.”
The paper relies on this without support.
Initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility.
“Given that initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Self-Refine is effective but is limited to single-generation reasoning tasks.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [15]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, this approach is suboptimal as it lacks problem-specific modeling, that is it does not recognize the constraints to be followed, and only prompts the model to evaluate and improve its responses without clear guidance on what to focus on.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Retry-based refinement methods Madaan et al. [2023], Gou et al. [2024] often discard the entire context and regenerate from scratch, losing correct intermediate work.”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Madaan et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“They unfairly instruct the models to generate initial responses that have problems intentionally, over-evaluating selfcorrection performance.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research has shifted toward enabling agents to autonomously self-improve when provided with suitable interactive environments.”
From A Survey of Large Language Models (2026) · cites this paper as [277]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We evaluate FACTOOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023;Chen et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The original prompt in Madaan et al. (2023) (Figure 7) does not clearly specify that the LLM needs to include all concepts in the prompt”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Yet a sobering empirical pattern has emerged: most autonomous self-improvement loops exhibit rapid asymptotic saturation. For instance, the Self-Refine paradigm plateaus after a mere few iterations [7].”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“while self-refinement(Madaan et al., 2023) and verification loops (Tang et al., 2023) iteratively critique and revise generations, improving factuality at the cost of higher inference overhead and a risk of over-editing.”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, unlike approaches such as Refine (Madaan et al., 2023), which struggle when combined with existing models on complex tasks”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some of the benchmarks (e.g. HotPotQA [18], GSM-8k [29], ) used in evaluations of self-verification are easy-that is, SoTA LLM performance is already high-and are therefore much less informative about the effects of the refinement procedure.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The self-refine approach was configured with four iterative refinement rounds at a temperature of 0.7, as adopted in prior research22.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Failed attempts and suboptimal decisions may accumulate, while new evidence from testing can invalidate earlier assumptions [5, 24, 33].”
From Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“existing text optimizers are poorly matched to this setting because they compress feedback too aggressively: they are memoryless, condition only on scalar scores, or restrict feedback to short templates or summaries.”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The reflection process is designed to enhance the system's capability to learn from mistakes within trajectories rather than only successes (Madaan et al., 2023; Shinn et al., 2023).”
From R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior self-reflection methods [11, 12] generate critiques of the LLM itself based on previous trial trajectories, leveraging past mistakes to guide the next trial.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While Self-Refine uses the model itself as its critic without additional context, our DECRIM pipeline employs a critic with fine-grained evaluation, assessing each constraint individually.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“But many works in this direction, such as Branch-SolveMerge (Saha et al., 2024) and Self-Refine (Madaan et al., 2023), assume constraint independence or focus on specific constraint types, limiting their applicability to real-world use cases.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“prior works have demonstrated the successful usage of self-correction that aligns model responses with specific preferences, such as altering the style of responses or enhancing their safety (Bai et al., 2022; Ganguli et al., 2023; Madaan et al., 2023).”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, systems like Voyager (Wang et al., 2023a), Self-Refine (Madaan et al., 2023), and SwarmAgentic (Zhang et al., 2025) employ profile-like adaptation and iterative self-improvement, but remain task-specific.”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some frameworks already experiment with using feedback from assertions or user grades to refine prompts [35, 39, 48].”
From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023;Chen et al., 2023c;Lu et al., 2023;Kim et al., 2023).”
From Large language models as tool makers (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The generate-test-repair loop is Reflexion-style iteration [Shinn et al., 2023, Madaan et al., 2023, Chen et al., 2024], and the symbolic analogue of learning an action model inside a planning loop is classical action-model learning [Lamanna et al., 2021, Yang et al., 2007, Aineto et al., 2019].”
From OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3 (2026) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“although self-correction research mostly focuses on reasoning tasks such as math reasoning and coding (Madaan et al., 2023;Gou et al., 2024;Huang et al., 2024a).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The few-shot examples used by Madaan et al. (2023) for generating initial responses unfairly have different properties from the target task.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In specific, [62] proposes a self-refine mechanism.This mechanism consists of three crucial components: output, feedback, and refinement.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“thus, they show that self-correction improves task performance by asking the model to identify missing concepts and then guiding it to incorporate these concepts through feedback.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Self-reflection and iterative refinement methods (Madaan et al., 2023; Mousavi et al., 2023; Han et al., 2025c) naturally align with short-term, inside-trial memory, as the agent repeatedly leverages intermediate reasoning traces or outcomes from prior attempts within the same task.”
From Memory in the Age of AI Agents (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While effective, these approaches add an additional layer of complexity, requiring the model to both recognize its own errors and articulate useful self-correction strategies.”
From Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most of them prompt or train language models to correct their initial results.”
From CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“various feedback and self-correction techniques have been introduced to improve LLM reliability, such as Reflexion and Self-Refine [21,22].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“Refinement -where solutions are instead critiqued and improved upon during resamplingoffers a possible avenue for breaking out of the aggregation rut.”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“MAGICORE also outperforms a combination of Self-Refine (Madaan et al., 2023) and Self-Consistency by 4.0%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-refinement methods (Madaan et al., 2023) show that models can improve through iterative critique or feedback, often using instructive signals or revision exemplars.”
From GIER: Gap-Driven Self-Refinement for Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“It has been demonstrated that prompting-only setups can work well for stylistic or qualitative improvements (e.g.Madaan et al., 2023;Chen et al., 2023b), but would require external feedback for reasoning tasks (Huang et al., 2023;Shinn et al., 2023;Kim et al., 2023;Madaan et al., 2023).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023), as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Meta-Harness is also closely related to methods such as ProTeGi, TextGrad, OPRO, GEPA, AlphaEvolve/OpenEvolve, and Feedback Descent, which iteratively improve prompts or other text artifacts using feedback from prior attempts [38; 31; 53; 51; 1; 35; 43; 26].”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“Building on early research showing that LLMs can produce useful feedback and iteratively improve outputs (Madaan et al., 2023), recent work has explored using multimodal models as visual critics to ground code generation and refinement.”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-Refine (Madaan et al., 2023) shows diminishing marginal returns across multiple rounds of self-refinement, with each additional iteration providing smaller gains.”
From Can LLMs Self-Correct Table Reasoning Errors? (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-Refine introduced iterative self-critique, demonstrating that models could improve their own outputs through multi-turn prompting loops [Madaan et al., 2023].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-correction methods enhance output quality through iterative refinement [6]”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“We observe only marginal improvements on REALINSTRUCT and a performance drop on IFEval using the Self-Refine (Madaan et al., 2023) pipeline, highlighting the limitations of traditional self-correction approaches for the multi-constrained instruction following task.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023)”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“and applying their self-correction prompt on top of model responses from our stronger version of the standard prompting again leads to a decrease in performance.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Table 8 demonstrates that our new prompt, denoted as Standard Prompting (ours) , significantly outperforms the results after self-correction of Madaan et al. (2023)”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“we take the Constrained Generation task in Madaan et al. (2023) as an example, where the task requires models to generate coherent sentences using all 20-30 input concepts.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Addressing the notable challenges of self-correction in large language models [Madaan et al., 2023, Huang et al., 2023b], AutoMix employs a non-LLM setup for routing and avoids escalating issues such as hallucination and reasoning errors [Dziri et al., 2023].”
From AutoMix: Automatically Mixing Language Models (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“First, 'self-improvement' is not one thing. The term is used for inference-time output revision (Self-Refine [6]), training loops on self-generated data (STaR [7], Self-Rewarding LMs [8]), agents that rewrite their own code (Gödel Agent [9], Darwin Gödel Machine [10]), and systems that autonomously conduct AI research [11]. These differ enormously in ambition and risk profile, yet the literature's”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“A key feature of the Code agent is its self-refinement mechanism [31], which enables iterative improvement of generated code.”
From SafeCodeRL: Security-Constrained Multi-Agent Reinforcement Learning for Trustworthy LLM-Generated IoT/CPS Software (2026) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“many recent studies have investigated the potential of LLMs to correct and refine their outputs, demonstrating that they indeed possess those capabilities (Welleck et al., 2023; Madaan et al., 2023; Shridhar et al., 2023; Ganguli et al., 2023; Wang et al., 2023b; Qi et al., 2023; Yang et al., 2023).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Similar to humans, LLMs do not always generate optimal outputs on their first attempt.”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Reflection-based methods, such as Reflexion [11] and Self-Refine [12], provide critical feedback that allows agents to learn from previous mistakes.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“Critique-and-refinement is a technique to improve the initial output of an LLM on a specific task.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies demonstrated that LLMs can serve as effective critics, generating detailed and constructive feedback [13, 14].”
From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Prompts used by Madaan et al. (2023) for generating initial responses instruct to generate responses that are not interesting and not very engaging, which are contradicting to the task goal.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, few existing approaches jointly evolve the full set of editable components [16]; most focus on a single component, typically the prompt [32, 50, 20], skills [19, 43], or an in-context playbook [49].”
From Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredIn the citing paper
“Prior studies often simply state their research questions as whether LLMs can self-correct their mis-takes (e.g., Kim et al., 2023;Madaan et al., 2023).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“Few-shot examples for generating initial responses include examples in settings different from the target task (positive to negative), while all few-shot examples for refinement are positive to negative.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“However, most studies in intrinsic self-correction (Madaan et al., 2023;Huang et al., 2024a) use simple prompts for generating feedback, and there is room for improvement.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“As more obvious weak prompts, Self-Refine (Madaan et al., 2023) uses instructions or few-shot examples that do not correctly correspond to the target task only for initial response generation (e.g., providing wrong target labels in few-shot examples), while using appropriate instructions for self-correction, as shown in Table 9 and 10.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“For example, the constrained generation task evaluated in Self-Refine (Madaan et al., 2023) is a task to generate a sentence that includes five specified words.We can easily evaluate the correctness by checking whether the five words are included in the generated sentence.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room?
“| This work | SELF-REFINE w/ GPT-3.5 SELF-REFINE w/ ChatGPT | 62.4 75.1 | | | SELF-REFINE w/ GPT-4 | 94.5 |”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Math Reasoning 64.1 64.1 (0) 74.8 75.0 (↑0.2) 92.9 93.1 (↑0.2)”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“| SELF-REFINE wins | 36.0 | 48.0 | 54.0 | | INIT wins | 23.0 | 18.0 | 16.0 | | Both are equal | 41.0 | 50.0 | 30.0 |”
Reported 100 · recomputed 116
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“* Well-known: CSCLE is not a well-known acronym.5/5 * Total score: 23/25”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
Reported 23 · recomputed 24
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“We use greedy decoding with a temperature of 0.7 for all setups.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
“We consider the temperature of both T = 0.0 (greedy) and T = 0.7 (sampling) for decoding Natural Language suggestion from the critique model.We always use a temperature T = 0.0 (greedy) when decoding Programming Language from the code editor.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
“Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINEand ChatGPT+SELF-REFINE across all tasks”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Sentiment Reversal 8.8 30.4 (↑21.6)11.4 43.2 (↑31.8)3.8 36.2 (↑32.4)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“* Well-known: CSCE is not a well-known acronym.2/5 * Total score: 20/25”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
Reported 20 · recomputed 21
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Refine Feedback
“We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo;OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Dialogue Response 36.4 63.6 (↑27.2) 40.1 59.9 (↑19.8)25.4 74.6 (↑49.2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“The FEEDBACK-REFINE iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
“Due to budget constraints, we run SELF-REFINE for N = 5 iterations.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
“Table 13: SELF-REFINE results from table 1 with Wilson confidence interval (at 95% confidence interval) and statistical significance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
“Table 13 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at α= 99% confidence interval) and statistical significance.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2×
“We also experiment with Vicuna-13b (Chiang et al., 2023), a version of LLaMA-13b (Touvron et al., 2023) fine-tuned on conversations sourced from the web.”
BackgroundMethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481 · cited 11×
“Dataset: (Zhang et al., 2015) 1000 review passages x: The food was fantastic.”
BackgroundMethodInfluentialThe Architecture of Complexity
Simon, H. A. (1991) · Springer eBooks · doi:10.1007/978-1-4899-0718-9_31 · Cited by 4,479 · cited 1×
“Iterative self -refinement is a fundamental characteristic of human problem-solving (Simon, 1962; Flower and Hayes, 1981; Amabile, 1983).”
BackgroundChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Interval Estimation for a Binomial Proportion
Brown, L. D., Cai, T., & Dasgupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,746 · cited 1×
“Table 13 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at α= 99% confidence interval) and statistical significance.”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 4×
“5 (text-davinci-003 and gpt-3.5-turbo; OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
BackgroundInfluentialEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Research works printed but not linked: 3
- Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu. 2019. Teaching a black-box learner. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1547-1555. PMLR.
- William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022a. Self-critiquing models for assisting human evaluators.
- Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022a. Peer: A collaborative language model. · DOI · Paper page
Cited by
Sorted by citations
Items 201–300 of 778
Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Yan, H., Su, M.-L., Zhang, H., Li, Z., Zhang, C., Zhang, S., Chen, Y., Bai, L., & Hu, S. (2026) · arXiv · doi:10.48550/arxiv.2609.01481 · Cited by 5
A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis
Spieser, J., Balapour, A., Meller, J., Patra, K. C., & Shamsaei, B. (2026) · Methods and Protocols · doi:10.3390/mps9020033 · Cited by 5
FactualRAG: a RAG-Powered framework for LLM fact-checking via IFCN-Accredited sources
Chang, L.-J., & Lai, C.-M. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01041-y · Cited by 5
“In addition, our runtime analysis contributes to ongoing discussions on the accuracy–efficiency trade–off in large language model reasoning systems (Madaan et al., 2023; Wang et al., 2022).”
Wang, Z., Moriyama, S., Wang, W.-Y., Gangopadhyay, B., & Takamatsu, S. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3676025 · Cited by 43
Leveraging Large Language Models for Institutional Portfolio Management: Persona-Based Ensembles
Abe, Y., Matsuo, S., Kondo, R., & Hisano, R. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825362 · Cited by 8
“15 Self-refine [28]: make the LLM provide feedback about itself and iteratively refine its output.”
Reflective prompt engineering: a new strategy for automated short answer scoring in biology
Ariely, M., Salman, A., Yarden, A., & Alexandron, G. (2025) · International Journal of Science Education · doi:10.1080/09500693.2025.2523571 · Cited by 7
“Recent research has shown that the ‘reasoning capabilities’ of LLMs can be improved through self-reflection, in which the LLM is prompted to reflect on its response to identify and correct potential mistakes without any external guidance (Chen et al., 2023; Madaan et al., 2024).”
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Singh, C., Morris, J. X., Rush, A. M., Gao, J., & Deng, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.384 · Cited by 18
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models
Farinhas, A., de Souza, J. G. C., & Martins, A. F. T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.733 · Cited by 14
Chen, P., & Li, Z. (2025) · Electronics · doi:10.3390/electronics14081662 · Cited by 4
Blazevičs, J. R., Ņikiforova, O., & Pastor, Ó. (2025) · Annals of Computer Science and Information Systems · doi:10.15439/2025f8736 · Cited by 5
“ISO 21502:2020, Project, programme and portfolio management — Guidance on project management, 2020 [35]; 4.”
RWTH-DBIS at LLMs4OL 2024 Tasks A and B
Peng, Y., Mou, Y., Zhu, B., Sowe, S. K., & Decker, S. (2024) · Open Conference Proceedings · doi:10.52825/ocp.v4i.2491 · Cited by 6
“One potential strategy involves incorporating feedback loops to continuously optimize prompts based on model outputs [29].”
BackgroundFlows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 34
Cohen, Y., & Aperstein, Y. (2024) · Preprints.org · doi:10.20944/preprints202405.0720.v1 · Cited by 5
R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory
Huang, T., Basu, K., Abdelaziz, I., Kapanipathi, P., May, J., & Chen, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1464 · Cited by 4
“The reflection process is designed to enhance the system’s capability to learn from mistakes within trajectories rather than only successes (Madaan et al., 2023; Shinn et al., 2023).”
Improving medication error classification using a reasoning large language model
Krifors, A., Beskow, T., Jonsson, M., Lindner, K.-J., Calås, J., Arwsbo, V., Sandström, K., & Norström, C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag004 · Cited by 5
“Moreover, iterative refinement methods, wherein the model re-evaluates and improves its outputs, have also shown promising results (11).”
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 4
Nam, Y., Jeong, J., & Kim, N. (2026) · arXiv · doi:10.48550/arxiv.2606.09500 · Cited by 4
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4
Self-citationZhang, C., Zhu, S., Yang, X., Tseng, Y.-C., Jiang, S., & Rzeszotarski, J. M. (2025) · Proceedings of the 7th ACM Conference on Conversational User Interfaces · doi:10.1145/3719160.3736618 · Cited by 4
Thought graph traversal for test-time scaling in chest X-ray VLLMs
Yao, Y., Wen, Z., Tong, Y., Tian, X., Li, X., Ma, X., Xu, D., & Gedeon, T. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113639 · Cited by 8
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Qin, Y., Wei, B., Ge, J., Kallidromitis, K., Fu, S., Darrell, T., & Wang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37359-5_25 · Cited by 79
A Survey of Progress in LLM Alignment From the Perspective of Reward Design
Ji, M., Wu, Y., Wu, Z., Wang, S., Yang, J., Dras, M., & Naseem, U. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3654921 · Cited by 24
Qu, M., Hu, Y., Han, K., Wei, Y., & Zhao, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00855 · Cited by 3
Li, S., Kallidromitis, K., Gokul, A., Koneru, A., Kato, Y., Kozuka, K., & Grover, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01453 · Cited by 29
Wu, Q., Zhang, W., Fan, P., Wang, K., Fan, Q., Chen, W., Mao, G., & Letaief, K. B. (2026) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2026.3728865 · Cited by 6
Xia, S.-J., Zhang, H., Jiang, Y., Chen, X., Chen, Y., Li, Z., Wan, Z., & Sun, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5731427 · Cited by 3
Human-AI Collaboration for Business Process Modeling with Petri Nets
Schlösser, T., Forell, M., Schüler, S., & Fritsch, A. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-95397-2_4 · Cited by 3
Abdelkarim, S., Lu, D., Flores, D., Jaeggi, S. M., & Baldi, P. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100170 · Cited by 6
“We built an LLM agent that consisted of three components that interact with each other, an actor, a critic, and a self-reflector, similar to the setups in Chen, Lin, Schärli, and Zhou (2024), Madaan et al. (2023), Shinn, Cassano, Gopinath, Narasimhan, and Yao (2023).”
AgentArcEval: An architecture evaluation method for foundation model based agents
Lu, Q., Zhao, D., Liu, Y., Zhang, H., Zhu, L., Xu, X., Shi, A., Tan, T., & Kazman, R. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112656 · Cited by 5
Wada, A., Nishizawa, M., YAMAMOTO, A., Akashi, T., Hagiwara, A., Irie, R., Hayakawa, Y. K., Kikuta, J., Shimoji, K., Sano, K., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-36904-5 · Cited by 4
Crystal: Introspective Reasoners Reinforced with Self-Feedback
Liu, J., Pasunuru, R., Hajishirzi, H., Choi, Y., & Çelikyılmaz, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.708 · Cited by 4
It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset
Kim, J., Kim, Y., Baek, I., Bak, J., & Lee, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.927 · Cited by 7
“Recent studies (Huang et al., 2022; Wang et al., 2022; Madaan et al., 2023) indicate that large language models can improve reasoning capabilities autonomously without supervised data, surpassing current techniques in accuracy.”
BackgroundPrompt Engineering ChatGPT for Codenames
Sidji, M., & Stephenson, M. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645591 · Cited by 4
“These ask models to follow a similar process of self critique or explanation and further improve performance for certain tasks compared to CoT prompting [15], [21].”
MethodDiverse AI Feedback For Large Language Model Alignment
Yu, T., Lin, T.-E., Wu, Y., Yang, M., Huang, F., & Li, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00746 · Cited by 12
Barrientos-Espillco, F., Pájares, G., Orozco, J. A. L., & Besada-Portas, E. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128169 · Cited by 4
Exploring the Potential of Anomaly Detection Through Reasoning with Large Language Models
Park, S., & Choi, D. (2025) · Applied Sciences · doi:10.3390/app151910384 · Cited by 3
Shen, S., Xiao, Z., & Han, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131859 · Cited by 4
Process Knowledge Graphs (PKG): Towards unpacking and repacking AI applications
Daga, E. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100846 · Cited by 4
Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math ProblemsPreprint
Ye, T., Xu, Z., Li, Y., & Allen-Zhu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5250631 · Cited by 9
Reconstruction of Landslide Events Using Llm-Based Agentic Ai with Multimodal DataPreprint
Pang, H., Lo, M. K., Leung, Y. F., & Wu, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5340542 · Cited by 4
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
“But many works in this direction, such as Branch-Solve-Merge (Saha et al., 2024) and Self-Refine (Madaan et al., 2023), assume constraint independence or focus on specific constraint types, limiting their applicability to real-world use cases.”
InfluentialBai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., & Pei, J. (2026) · arXiv · doi:10.48550/arxiv.2604.22750 · Cited by 3
“Prior work on self-feedback (Madaan et al., 2023) and language model calibration (Kadavath et al., 2022) provides theoretical grounding for the broader claim that models can reason about their own outputs and uncertainty.”
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“These take many forms: common search algorithms with the LLM cast in the role of heuristic [37, 11], approaches which reduce error rates by enforcing various consistency conditions [8, 6, 17], and direct LLM self-critique [27, 35, 5, 15, 21, 38].”
BackgroundMethodMeta-Harness: End-to-End Optimization of Model Harnesses
Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026) · arXiv · doi:10.48550/arxiv.2603.28052 · Cited by 3
“…to harness engineering because they typically operate with short-horizon or heavily compressed feedback: some condition only on the current candidate [31; 51; 53], others rely primarily on scalar scores [35; 12], and others restrict feedback to short templates or LLM-generated summaries [1; 26].”
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3
AgentSquare: Automatic LLM Agent Search in Modular Design Space
Shang, Y., Li, Y., Zhao, K., Ma, L., Liu, J., Xu, F., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2410.06153 · Cited by 3
Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3729965 · Cited by 3
User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal
Liu, Y., Zhang, M. J. Q., & Choi, E. (2025) · arXiv · doi:10.48550/arxiv.2507.23158 · Cited by 3
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3
Leveraging Large Language Models to Detect npm Malicious Packages
Zahan, N., Burckhardt, P., Lysenko, M., Aboukhadijeh, F., & Williams, L. (2024) · arXiv · doi:10.48550/arxiv.2403.12196 · Cited by 3
Bogdanov, I., Lung, C.-H., Kunz, T., Gao, J., Taylor, A., & Zaman, M. (2026) · Proceedings of the ACM Conference on AI and Agentic Systems · doi:10.1145/3786335.3813149 · Cited by 3
Zhang, J., Yuan, B., & Zhang, Q. (2026) · Entropy · doi:10.3390/e28090951 · Cited by 2
Xia, C., Gou, T., Xu, Y., et al. (2026) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2026.3672814 · Cited by 2
Video-T1: Test-Time Scaling for Video Generation
Liu, F., Wang, H., Cai, Y., Zhang, K., Zhan, X., & Duan, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01735 · Cited by 30
Is your prompt poisoning code? Defect induction rates and security mitigation strategies
Wang, B., Zhong, Y., Wan, M., Yu, W., Ouyang, Y., Huang, Y., Wu, H., & Li, H. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10866-8 · Cited by 2
Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement
Jin, H., & Chen, H. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00638-5 · Cited by 13
An approach to optimizing semantic consistency for text-to-digital human generation
Liu, X., Liu, X., Yang, P., Wang, Z., & Liu, F. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111909 · Cited by 4
Learning to Verify and Assure Cyber-Physical Systems
Jin, H., Zhang, T., Ramamurthy, A., Hamza, A., & Malinoski, M. (2024) · AIAA SciTech Forum · doi:10.2514/6.2024-1853 · Cited by 3
G, M., K., N. R., & Marimuthu, M. (2023) · 2023 Innovations in Power and Advanced Computing Technologies (i-PACT) · doi:10.1109/i-pact58649.2023.10434839 · Cited by 4
“A novel method called SELF-REFINE [12] for object detection is proposed, which leverages self-supervision and iterative refinement with self-feedback to improve the accuracy of object detection models.”
MethodTranslate-and-Revise: Boosting Large Language Models for Constrained Translation
Huang, P., Mu, Y., Wu, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_8 · Cited by 2
SGEU: enhancing LLM reasoning via backward exemplar generation and verification
Wang, Z., Zhou, X., Yang, Y., Ma, B., Wang, L., & Dong, R. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06529-8 · Cited by 1
Self-citationTRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Xiong, J., Shen, J., Yuan, Y., Wang, H., Yin, Y., Liu, Z., Li, L., Guo, Z., Cao, Q., Huang, Y., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.711 · Cited by 2
ExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook
Sun, W., Yu, X., He, S., Zhao, J., & Liu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1034 · Cited by 4
Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task
Du, C., Tian, J., Liao, H., Chen, J., He, H., & Jin, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.150 · Cited by 2
AI-Assisted Identification of Policy-Salient Research Priorities and Emerging IssuesPreprint
Rudd, M. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4502450 · Cited by 2
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
Symbolic Learning Enables Self-Evolving AgentsPreprint
Zhou, W., Ou, Y., Ding, S., Li, L., Wu, J., Wang, T., Chen, J., Wang, S., Xu, X., Zhang, N., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5379114 · Cited by 6
Vision-Guided Iterative Refinement for Frontend Code Generation
Sansford, H., Law, D. H. C., Liu, W., Tripathi, A., Agarwal, N., & Burg, G. J. J. V. D. (2026) · arXiv · doi:10.48550/arxiv.2604.05839 · Cited by 2
“Building on early research showing that LLMs can produce useful feedback and iteratively improve outputs (Madaan et al., 2023), recent work has explored using multimodal models as visual critics to ground code generation and refinement.”
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3
“In the two running examples (and in Yang et al. (2023); Madaan et al. (2023); Shinn et al. (2023)), the feedback loop induces optimization by iteratively refining parts of the world-LLM system.”
BackgroundBALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2
“Advanced Reasoning Strategies Beyond simply prompting LLMs and VLMs to directly predict the next action of game-play, BALROG also supports the study of more advanced reasoning techniques like chain-of-thought (Wei et al., 2022b), self-refinement (Madaan et al., 2024), and basic planning.”
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2024) · arXiv · doi:10.48550/arxiv.2410.04444 · Cited by 2
“…(Wei et al., 2022) 64.2 ± 0.9 28.0 ± 3.1 65.4 ± 3.3 29.2 ± 3.1 COT-SC (Wang et al., 2023b) 64.4 ± 0.8 28.2 ± 3.1 65.9 ± 3.2 30.5 ± 3.2 Self-Refine (Madaan et al., 2024) 59.2 ± 0.9 27.5 ± 3.1 63.5 ± 3.4 31.6 ± 3.2 LLM Debate (Du et al., 2023) 60.6 ± 0.9 39.0 ± 3.4 65.6 ± 3.3 31.4 ± 3.2…”
InfluentialThink before you speak: Training Language Models With Pause Tokens
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., & Nagarajan, V. (2023) · arXiv · doi:10.48550/arxiv.2310.02226 · Cited by 2
Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback
Potamitis, N., & Arora, A. (2025) · arXiv · doi:10.48550/arxiv.2504.12951 · Cited by 2
SkillOpt: Executive Strategy for Self-Evolving Agent Skills
Yang, Y., Gong, Z., Huang, W., Yang, Q., Zhou, Z., Huang, Z., Li, Y., Gao, X., Dai, Q., Liu, B., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.23904 · Cited by 2
Self-citationAdversarial Review: Structured Disagreement for Grounded Agentic Code Review
Qiu, E. S., & Gill, J. (2026) · arXiv · doi:10.48550/arxiv.2608.18167 · Cited by 2
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
Singhvi, A., Shetty, M., Tan, S., Potts, C., Sen, K., Zaharia, M., & Khattab, O. (2023) · arXiv · doi:10.48550/arxiv.2312.13382 · Cited by 2
Wang, Z., Chen, J., Wei, Z., Lin, L., & Tong, G. (2026) · Sensors · doi:10.3390/s26113502 · Cited by 2
The economics of accuracy for medical reasoning with large language models
Bhattacharyya, K., & Kamabattula, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001182 · Cited by 1
Fairbanks, J., & Serra, E. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401712 · Cited by 1
Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Models
Huang, L., Li, D. B., Liu, H., & Cheng, L. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11400862 · Cited by 12
Advancing the science of qualitative patient preference assessment using large language models
Grover, T., Krebs, E., Weymann, D., Ehman, M., & Regier, D. A. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001263 · Cited by 1
VFlow: Discovering Optimal Agentic Workflows for Verilog Generation
Wei, Y., Huang, Z., He, L., Huang, L., Lin, T.-J., & Xing, W. W. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420713 · Cited by 3
MedSegAgent: A Universal and Scalable Multi-Agent System for Instructive Medical Image Segmentation
Huang, Z., Wang, H., Ye, J., Ji, Y., Hu, X., Liu, L., Yang, Z., Li, W., Hu, M., Su, Y., et al. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3677444 · Cited by 2
A neural-symbolic AI agent system for biomedical concept mapping
Zhang, G., Fang, Y., Chen, F., Ta, C., Hripcsak, G. M., Ryan, P., Peng, Y., & Weng, C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02594-6 · Cited by 1
Wang, Y., & Ge, Y. (2026) · Information Systems Research · doi:10.1287/isre.2024.1518 · Cited by 1
VSP: Diagnosing the Dual Challenges of Perception and Reasoning in Spatial Planning Tasks for MLLMS
Wu, Q., Zhao, H., Saxon, M., Bui, T., Wang, W. Y., Zhang, Y., & Chang, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00219 · Cited by 1
Reconstruction of landslide events in urban setting using LLM-based Agentic AI with multimodal data
Pang, H., Lo, M. K., Leung, Y. F., & Wu, S. (2026) · SOILS AND FOUNDATIONS · doi:10.1016/j.sandf.2026.101789 · Cited by 6
Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder
Dang, J., Wu, X., Yan, H., Zheng, H., Zheng, W., Lai, J., Hu, B., & Chua, T. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3696843 · Cited by 5
InteriorAgent: LLM Agent for Interior Design-Aware 3D Layout Generation
Gupta, K., Mehta, I., Wang, K., Chua, N., Krishna, A., Deng, Y., Ramamoorthi, R., & Chandraker, M. K. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00138 · Cited by 2
Mo, M., Zhang, H., Tan, Y., Huang, B., & Li, C. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116598 · Cited by 1
El agente cuántico: automating quantum simulations
Gustín, I., Calderón, L., Pérez-Sánchez, J. B., Crebolder, C., Gonthier, J. F., Vakili, M. G., Nakamura, Y., Panicker, K., Ramprasad, M., Yin, A., et al. (2026) · Reports on Progress in Physics · doi:10.1088/1361-6633/ae8933 · Cited by 16
Yuan, M., Wang, T., & Qian, Y. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01028-3 · Cited by 3
Taricco, G. (2026) · Entropy · doi:10.3390/e28070811 · Cited by 1
Can large language models be a cardinality estimator? An empirical study
Liu, L., Wu, Y., Wang, Y., Chang, Z., Su, R., Wu, P., Chen, J., Jiang, F., Shi, R., Cui, B., et al. (2026) · The VLDB Journal · doi:10.1007/s00778-026-00975-7 · Cited by 1
Choi, H., & Ha, J.-E. (2026) · International Journal of Precision Engineering and Manufacturing · doi:10.1007/s12541-026-01548-y · Cited by 1
Destroy-repair of reasoning chains via adaptive large neighborhood search in small language models
Qasim, K. U., Zhang, J., Aslam, M. W., et al. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123959 · Cited by 1
-, S. B., Ranjan, S., Adhitya, M., Suri, V., Hassija, V., Chamola, V., & Hussain, A. (2026) · Cognitive Computation · doi:10.1007/s12559-026-10619-1 · Cited by 5
Yu, H., Iqbal, S., Neto, E. C. P., Buffett, S., & Taylor, A. (2026) · International Journal of Information Security · doi:10.1007/s10207-026-01321-2 · Cited by 1
Large Language Models for UAV Autonomy from a Perception–Cognition–Action Perspective
Xiong, T., Zhan, J., Deng, Q., Wang, X., Fan, C., Liu, X., & Zhang, T. (2026) · Drones · doi:10.3390/drones10090669 · Cited by 1
Metadata
- Authors
- Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 46534-46594
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.17651
- OpenAlex
- W7133236450
- Cited by
- 2,314
- References
- 52
- Citations by year
- 2026: 1329 · 2025: 460 · 2024: 338 · 2023: 186
- FWCI
- 260.57
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- iterative refinement, self-feedback, large language models, mathematical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
