The DOI matches its DataCite record; 47 of the 53 references it cites were found in a registry or an index; 12 of the 22 sources its claims cite were checked for support.
Self-Refine: Iterative Refinement with Self-Feedback
Open accessAbstract
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
SELF-REFINE improves large language model outputs by iteratively generating self-feedback and refining the same model’s output without supervised training, additional training, or reinforcement learning. Across 7 diverse tasks, it improves over one-step generation by ∼20% absolute on average, with especially large gains on preference-based tasks.
Key findings
- SELF-REFINE consistently improved GPT-3.5, ChatGPT, and GPT-4 across all evaluated tasks. Examples include GPT-4 Dialogue Response Generation improving from 25.4 to 74.6 (+49.2), GPT-4 Code Optimization from 27.3 to 36.0 (+8.7), and GPT-4 Constrained Generation from 15.0 to 45.0 (+30.0).
- Feedback quality matters: in Code Optimization, SELF-REFINE feedback scored 27.5 vs. 26.0 for generic feedback and 24.8 for no feedback; in Sentiment Reversal, 43.2 vs. 31.2 vs. 0; in Acronym Generation, 56.4 vs. 54.0 vs. 48.0.
- Iterative refinement generally improves outputs, with diminishing returns: Code Optimization rose from 22.0 at y0 to 28.8 at y3; Sentiment Reversal from 33.9 to 36.8; Constrained Generation from 29.0 to 49.7.
- Math Reasoning gains were modest: GPT-3.5 64.1 to 64.1 (0), ChatGPT 74.8 to 75.0 (+0.2), GPT-4 92.9 to 93.1 (+0.2). With oracle feedback, GPT-3 improved by 4.8% and GPT-4 by 0.7%.
Methods
- SELF-REFINE alternates between FEEDBACK and REFINE using the same model M: initial output y0 = M(pgen ∥ x); feedback fbt = M(pfb ∥ x ∥ yt); refinement yt+1 = M(prefine ∥ x ∥ y0 ∥ fb0 ∥ … ∥ yt ∥ fbt). It iterates until a stopping condition or up to a maximum of 4 iterations, using few-shot prompts and no supervised training, additional training, or reinforcement learning.
- Evaluation covered 7 tasks: Dialogue Response Generation, Code Optimization, Code Readability Improvement, Math Reasoning, Sentiment Reversal, Acronym Generation, and Constrained Generation (CommonGen-Hard). Base LLMs were GPT-3.5 (text-davinci-003), ChatGPT (gpt-3.5-turbo), GPT-4, and CODEX for code tasks. Metrics included task-specific metrics, human preference, and GPT-4 preference.
- Analysis included ablations on feedback quality and iteration count, comparison against generating multiple outputs, testing Vicuna-13B, and a qualitative error analysis of 70 samples for Code Optimization and Math Reasoning.
Limitations
- Stated: the base models need sufficient few-shot modeling or instruction-following abilities to provide feedback and refine in-context; Vicuna-13B struggled to generate feedback in the required format and often failed to refine properly.
- Stated: experiments used non-open-sourced models (GPT-3.5, ChatGPT, GPT-4, CODEX), whose pretraining corpus, sizes, and biases are not fully described, and which are not free to use; datasets were exclusively English; the approach does not explicitly guard against bad actors steering models toward toxic or harmful text.
- Evident from the text: human A/B evaluations were conducted by the authors, and GPT-4 was used as a proxy for human preference with reported correlations of 82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Generation. The paper also notes Math Reasoning gains are limited by the model’s inability to accurately identify errors, with ChatGPT feedback being “everything looks good” for 94% of instances.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
70 Own result · 25 Own interpretation · 9 No basis given · 7 Partly supported · 5 Own method · 4 Supported · 4 Sources not found · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“In code-generation tasks, SELF-REFINE improves the initial generation by up to absolute 13% when applied to strong code models such as Codex (code-davinci-002; Chen et al., 2021).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Refine Feedback
What the sources say
- Supportedcites Chen et al., 2021 source
The Chen et al. citation sits in the parenthetical identifying Codex, rather than after the reported SELF-REFINE improvement. Section 3, “Code Fine-Tuning” (p. 4), describes producing Codex and evaluating it on code-generation tasks, supporting that identification. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, these reinforcement learning methods require updating the model's parameters, unlike SELF-REFINE.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Supportedcites Stiennon et al., 2020 source
The Introduction (p. 2) says the policy is updated using PPO, and §3.4 (p. 6) describes a learned RL policy with parameters. This supports the claim about the reinforcement-learning methods discussed; the contrast with SELF-REFINE is the citing authors’ framing, not a finding attributed to this source. (LLM, conf=0.90)
- Not matchedcites Lu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Stiennon et al. (2020) source
The Introduction (p. 2) says the policy is trained through reinforcement learning and updated using PPO. Section 3.4 (p. 6) also describes the learned RL policy as having parameters. The comparison with SELF-REFINE is the citing authors’ own framing, not a claim this source is needed to establish. (LLM, conf=0.92)
- Not matchedcites Lu et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al. (2022a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This adjustment notably enhanced performance in the Math Reasoning task, with GPT-3 improving by 4.8% and GPT-4 by 0.7% Table 9.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
What the sources say
- Partly supportedcites Welleck et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.5 (pp. 8–9) reports that correcting only incorrect outputs and adding explicit feedback improve results in math program synthesis. It does not report the claimed 4.8% GPT-3 or 0.7% GPT-4 gains; its math results use different models and figures. The cited paper’s Table 9 concerns E2E hyperparameters, not those gains. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These methods differ from SELF-REFINE (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Partly supportedcites Stiennon et al., 2020 sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says this paper’s RL policy is updated using a reward for the entire generated summary, and a note on p. 6 explicitly says the reward model gives no rewards at intermediate time steps. That supports the distinction for this method, but not the broader claim that reinforcement-learning approaches generally cannot access feedback on intermediate generations. (LLM, conf=0.84)
- Not matchedcites Lu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked to generate a sentence containing up to 30 given concepts.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 4 demonstrates that on average, the quality of the output improves as the number of iterations increases.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all evaluated tasks, outputs generated with SELF-REFINE are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ∼20% absolute on average in task performance.”
- Also stated in: § 3.3 Results · plain
“SELF-REFINE consistently improves over base models across all model sizes, and additionally outperforms the previous state-of-the-art across all tasks.”
- Also stated in: § Refine Feedback · plain
“We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo;OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.3 Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, SELF-REFINE leads to especially high gains.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in the Code Optimization task, the initial output (y 0 ) has a score of 22.0, which improves to 28.8 after three iterations (y 3 ).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 35
- The quote only partly states this claim. 5
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“CommonGen-Hard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only the 3-5 related concepts given in CommonGen.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > K New Tasks
What the sources say
- Partly supportedcites Lin et al., 2020 sourceExperimental — accuracy not yet measured
Section 3.1 says CommonGen uses concept sets of three to five concepts, and Section 2 says the output should be a grammatical sentence using all given concepts. Our copy does not mention CommonGen-Hard or a requirement to incorporate 20–30 concepts (searched for “CommonGen-Hard” and “20-30 concepts”), so that part is unsupported. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing literature (Ouyang et al., 2022) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
What the sources say
- Partly supportedcites Ouyang et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.5 (page unavailable) calls the pretrained models’ behavior “poorly characterized,” which lends some support to the concern about incomplete characterization. But the paper does not say that existing literature omits the listed details; its Introduction gives the model sizes, and Section 5.3 discusses biased outputs. The claimed lack of disclosure about model sizes and biases is therefore not supported. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirically, we evaluated SELF-REFINE using the same base model of GPT-3 as Self-Correction, and with the same settings on the GSM8K benchmark.Self-Correction achieved 45.9% accuracy while SELF-REFINE (this work) achieved 55.7% (↑9.8).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Partly supportedcites Welleck et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) reports 45.92% accuracy on GSM with a GPT-3 Instruct generator and self-corrector, supporting the rounded 45.9% figure. The paper does not report SELF-REFINE’s 55.7% result or establish that the two methods used the same settings, so it does not substantiate the 9.8-point comparison. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Self-Correction trains a separate refiner (or”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Supportedcites Welleck et al., 2022 source
Section 2 (p. 2) says Self-Correction keeps the generator separate and trains the corrector with different objectives for different task requirements. The Introduction (pp. 1–2) also describes a trained corrector and its use across three tasks. The comparison describing SELF-REFINE’s prompting approach is not a claim about this source. (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Few-shot prompting (also referred to as”
What the sources say
- Supportedcites Brown et al., 2020 source
Section 2 Approach (p. 6) defines few-shot as giving the model K task examples as conditioning. It describes each example as a context and desired completion—the input-output pairs in the claim—followed by a final context for the model to complete. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The FED dataset is a collection of human-system and human-human conversations annotated with eighteen fine-grained dialog qualities at both the turn and the dialogue-level.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
What the sources say
- Supportedcites Mehri and Eskenazi, 2020 source
The Introduction describes the FED dataset as human quality annotations of human-system and human-human conversations. Section 3 (Data Collection) specifies eighteen fine-grained qualities: eight annotated at the turn level and ten at the dialogue level. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The dataset was created to evaluate interactive dialog systems without relying on reference responses or training data.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
What the sources say
- Partly supportedcites Mehri and Eskenazi, 2020 sourceExperimental — accuracy not yet measured
The Introduction says the FED dataset was collected for fine-grained evaluation of interactive dialog and contains no training data. It also says the dataset is intended to benchmark automatic metrics against human judgments; the absence of reference responses is a property of the FED metric, not a stated purpose or property of the dataset. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This effect is more pronounced in tasks like Sentiment Transfer, where changing from our feedback to generic feedback leads to a significant performance drop (43.2 to 31.2), and the task fails without feedback.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Code Optimization, performance slightly dips from 27.5 (SELF-REFINE feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINEand ChatGPT+SELF-REFINE across all tasks, even in tasks where the initial base results of GPT-4 were lower than GPT-3.5 or ChatGPT.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To address this, SELF-REFINE's feedback generates explicit numerical scores to capture the different aspects of output quality.This allows for a more balanced evaluation of outputs and the selection of the most appropriate one.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Vicuna-13b was able to consistently follow the task initialization prompt.However, it struggled to follow the prompts intended for feedback and refinement.This often led to outputs that resembled assistant-like responses, a representative example of which can be found in Appendix G.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > G Evaluation of Vicuna-13b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 12 breaks down errors found in the refinement stage of SELF-REFINE.Errors include: not being robust (10%), ignoring feedback (25%), and introducing a new problem (20%).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback and Refinement Error Analysis for Response Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that nearly all of GPT-4 gains are statistically significant, ChatGPT gains are significant for 4 out of 7 datasets, and GPT-3.5 gains are significant for 3 out of 7 datasets.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: GPT-4 is capable of assessing and providing reasoning for complex tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE demonstrates its iterative feedback and refinement capabilities in the context of website layout generation.ChatGPT initially produces a rudimentary layout for a given topic, and then uses the FEEDBACK to suggest specific, actionable improvements, as demonstrated in Figures 7 and 9.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > I Beyond Benchmarks
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Starting from a rudimentary initial design, SELF-REFINE refines HTML, CSS, and JS to evolve the website in terms of both usability and aesthetics.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Going Beyond Benchmarks
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the Dialogue Response Generation task, SELF-REFINE can ignore good feedback but in a majority of cases, it is robust to bad feedback and ignores bad feedback.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, SELF-REFINE achieves superior performance while using only 4 samples at most, significantly fewer than the 16 and 32 samples employed by other models.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > F Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance may not always monotonically increase with iterations: in multi-aspect feedback tasks like Acronym Generation, where the output quality can vary during iteration with improvement in one aspect but decline in another aspect.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 11 reports the occurrence of different types of errors in our sample, which includes Incorrect Feedback (25%), Generic Feedback (30%), and Incorrect Scoring (10%).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback and Refinement Error Analysis for Response Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A similar selection is possible for other tasks like Math Reasoning and Sentiment Reversal, while we observe that output quality increases monotonically with iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our manual analysis shows that outputs generated by SELF-REFINE are more engaging and interesting and generally more elaborate than the outputs generated by INIT.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE w/ GPT-3 | 55.7”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CommonGen-Hard requires models to generate coherent and grammatically correct sentences incorporating 20-30 concepts, as opposed to the original task which presents a set of 3-5 related concepts.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE wins | 36.0 | 48.0 | 54.0”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“INIT wins | 23.0 | 18.0 | 16.0”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE w/ GPT-4 | 94.5”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“PaL w/ GPT-4 | 93.3”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE -feedback | 1 | 10.1 | 62.15 | 3.03”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“SELF-REFINE | 1 | 15.3 | 59.64 | 2.90”
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the FEEDBACK step of SELF-REFINE.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For dramatic response evaluation, we found that the preference rate went down drastically to 58.92%, from 80.09%.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > O Math Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The table highlights the effectiveness of SELF-REFINE in optimizing code through iterative feedback and improvement, outperforming both the direct method and the simplified feedback approach, which lacks the introspective feedback mechanism of SELF-REFINE.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Direct | - | 9.7 | 62.29 | 3.09”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“SELF-REFINE | 2 | 15.6 | 65.60 | 3.74”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“PaL w/ ChatGPT | 74.2”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sentiment Transfer | 75.00 | 21.43 | 3.57”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“Acronym Generation | 44.59 | 12.16 | 43.24”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“Response Generation | 47.58 | 19.66 | 32.76”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Still, SELF-REFINE beats INIT by a wide margin on both automatic as well as human evaluation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At T = 0.0, SELF-REFINE produces less meaningful variables, less comments per line but even more function units.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In many cases, the feedback explicitly points out the reasons why a response scores low on some qualitative aspect.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At T = 0.7, SELF-REFINE produces more meaning variables, more function units and slightly more comments compared to the human annotators on average.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, this task has an extremely large number of reasonable outputs, and thus SELF-REFINE allows to better explore the space of possible outputs.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe that this task benefits significantly from SELF-REFINE because there are more opportunities to miss some of the concepts on the first attempt, and thus SELF-REFINE allows the model to fix these mistakes subsequently.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, the experiments in this work were performed with language models that are not open-sourced, namely GPT-3.5, ChatGPT, GPT-4, and CODEX.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Another limitation of our work is that we exclusively experiment with datasets in English.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE focuses on iterative creation with introspective feedback, making it suitable for evaluating the effectiveness of language models on the CommonGen-Hard task.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > K New Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This demonstrates the potential of SELF-REFINE in real-world, complex, and creative tasks.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Going Beyond Benchmarks
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Compared to a concurrent work, Reflexion (Shinn et al., 2023), our approach involves correction using feedback, whereas their setup involves finding the next best solution in planning using ReAct.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Not matchedcites Shinn et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the limited performance of Vicuna-13b suggests that this model may require more extensive prompt-engineering for optimal performance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > G Evaluation of Vicuna-13b
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, we release our code and model outputs to ensure the reproducibility of our work.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other languages, the current models may not provide the same benefits.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, having multiple FEEDBACK-REFINE iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This distinction allows our method to offer more precise and actionable feedback, making it suitable for a wider range of natural language generation tasks, including those that may not necessarily involve step-by-step planning such as open-ended dialogue generation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We thus hypothesize that since Vicuna-13B was trained on conversations, it does not generalize as well as instruction-based models to test-time few-shot tasks.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
What the sources say
- Not matchedcites Chiang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirectSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The setup was blind, i.e., the judges did not know which outputs were generated by which method.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > C Human Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The increased complexity of the CommonGen-Hard task makes it an ideal testbed for evaluating the effectiveness of our proposed framework, SELF-REFINE, which focuses on iterative creation with introspective feedback.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The judges were not aware of the method that generated each sample.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, acronym generation is a natural method testbed for our approach.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Acronym generation requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.
- Rests on an assumption: Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.
In the paper
“The primary objective of PIE is to optimize a given program by implementing algorithmic modifications that lead to improved runtime performance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
What the sources say
- Not matchedcites Madaan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through iterative creation and self-reflection, the SELF-REFINE framework empowers language models to progressively enhance the quality of their output, closely mimicking the human creative process and demonstrating its ability to improve generated text on complex and demanding natural language generation tasks like CommonGen-Hard (Figure 15).”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility.
In the paper
“Performance-Improving Code Edits or PIE (Madaan et al., 2023) focuses on enhancing the efficiency of functionally correct programs.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
What the sources say
- Not matchedcites Madaan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This significant increase in the number of concepts tests the model's ability to perform advanced commonsense reasoning, contextual understanding, and creative problem-solving, as it must generate meaningful sentences that encompass a broader range of ideas.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
Large language models often fall short in addressing intricate requirements.
“Although large language models (LLMs) can generate coherent outputs, they often fall short in addressing intricate requirements.”
The paper relies on this without support.
Math errors can be nuanced and sometimes limited to a single line or incorrect operation.
“In math, errors can be nuanced and sometimes limited to a single line or incorrect operation.”
The paper relies on this without support.
GPT-4 is capable of assessing and providing reasoning for complex tasks.
“In light of the impressive achievements of GPT-4 in assessing and providing reasoning for complex tasks, we leverage its abilities for evaluation in SELF-REFINE.”
The paper relies on this without support.
Code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase.
“though not related to the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase.”
The paper relies on this without support.
Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance.
“Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach.”
The paper relies on this without support.
Automated metrics may not provide an accurate assessment of text generation tasks.
“However, we acknowledge that automated metrics may not provide an accurate assessment of text generation tasks and rely on human evaluation instead.”
The paper relies on this without support.
Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics.
“Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics.”
The paper relies on this without support.
text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts, making GPT-3.5 a strong baseline.
“text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts and hence GPT-3.5 is a strong baseline.”
The paper relies on this without support.
Acronym generation requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.
“Like in email writing, acronym generation also requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.”
The paper relies on this without support.
Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.
“Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.”
The paper relies on this without support.
Some acronym quality criteria are difficult to quantify and are a matter of human preference.
“Some of these criteria are difficult to quantify, and are a matter of human preference.”
The paper relies on this without support.
Modern LLMs have superior instruction following capabilities.
“As with other modules, we leverage the superior instruction following capabilities of modern LLMs to instead provide a few demonstrations of each task.”
The paper relies on this without support.
Initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility.
“Given that initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Self-Refine is effective but is limited to single-generation reasoning tasks.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [15]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, this approach is suboptimal as it lacks problem-specific modeling, that is it does not recognize the constraints to be followed, and only prompts the model to evaluate and improve its responses without clear guidance on what to focus on.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Retry-based refinement methods Madaan et al. [2023], Gou et al. [2024] often discard the entire context and regenerate from scratch, losing correct intermediate work.”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Madaan et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“They unfairly instruct the models to generate initial responses that have problems intentionally, over-evaluating selfcorrection performance.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research has shifted toward enabling agents to autonomously self-improve when provided with suitable interactive environments.”
From A Survey of Large Language Models (2026) · cites this paper as [277]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We evaluate FACTOOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023;Chen et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The original prompt in Madaan et al. (2023) (Figure 7) does not clearly specify that the LLM needs to include all concepts in the prompt”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Yet a sobering empirical pattern has emerged: most autonomous self-improvement loops exhibit rapid asymptotic saturation. For instance, the Self-Refine paradigm plateaus after a mere few iterations [7].”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“while self-refinement(Madaan et al., 2023) and verification loops (Tang et al., 2023) iteratively critique and revise generations, improving factuality at the cost of higher inference overhead and a risk of over-editing.”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, unlike approaches such as Refine (Madaan et al., 2023), which struggle when combined with existing models on complex tasks”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some of the benchmarks (e.g. HotPotQA [18], GSM-8k [29], ) used in evaluations of self-verification are easy-that is, SoTA LLM performance is already high-and are therefore much less informative about the effects of the refinement procedure.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The self-refine approach was configured with four iterative refinement rounds at a temperature of 0.7, as adopted in prior research22.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Failed attempts and suboptimal decisions may accumulate, while new evidence from testing can invalidate earlier assumptions [5, 24, 33].”
From Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“existing text optimizers are poorly matched to this setting because they compress feedback too aggressively: they are memoryless, condition only on scalar scores, or restrict feedback to short templates or summaries.”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The reflection process is designed to enhance the system's capability to learn from mistakes within trajectories rather than only successes (Madaan et al., 2023; Shinn et al., 2023).”
From R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior self-reflection methods [11, 12] generate critiques of the LLM itself based on previous trial trajectories, leveraging past mistakes to guide the next trial.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While Self-Refine uses the model itself as its critic without additional context, our DECRIM pipeline employs a critic with fine-grained evaluation, assessing each constraint individually.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“But many works in this direction, such as Branch-SolveMerge (Saha et al., 2024) and Self-Refine (Madaan et al., 2023), assume constraint independence or focus on specific constraint types, limiting their applicability to real-world use cases.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“prior works have demonstrated the successful usage of self-correction that aligns model responses with specific preferences, such as altering the style of responses or enhancing their safety (Bai et al., 2022; Ganguli et al., 2023; Madaan et al., 2023).”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, systems like Voyager (Wang et al., 2023a), Self-Refine (Madaan et al., 2023), and SwarmAgentic (Zhang et al., 2025) employ profile-like adaptation and iterative self-improvement, but remain task-specific.”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some frameworks already experiment with using feedback from assertions or user grades to refine prompts [35, 39, 48].”
From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023;Chen et al., 2023c;Lu et al., 2023;Kim et al., 2023).”
From Large language models as tool makers (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The generate-test-repair loop is Reflexion-style iteration [Shinn et al., 2023, Madaan et al., 2023, Chen et al., 2024], and the symbolic analogue of learning an action model inside a planning loop is classical action-model learning [Lamanna et al., 2021, Yang et al., 2007, Aineto et al., 2019].”
From OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3 (2026) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior studies often simply state their research questions as whether LLMs can self-correct their mis-takes (e.g., Kim et al., 2023;Madaan et al., 2023).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“although self-correction research mostly focuses on reasoning tasks such as math reasoning and coding (Madaan et al., 2023;Gou et al., 2024;Huang et al., 2024a).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The few-shot examples used by Madaan et al. (2023) for generating initial responses unfairly have different properties from the target task.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In specific, [62] proposes a self-refine mechanism.This mechanism consists of three crucial components: output, feedback, and refinement.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“thus, they show that self-correction improves task performance by asking the model to identify missing concepts and then guiding it to incorporate these concepts through feedback.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Self-reflection and iterative refinement methods (Madaan et al., 2023; Mousavi et al., 2023; Han et al., 2025c) naturally align with short-term, inside-trial memory, as the agent repeatedly leverages intermediate reasoning traces or outcomes from prior attempts within the same task.”
From Memory in the Age of AI Agents (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While effective, these approaches add an additional layer of complexity, requiring the model to both recognize its own errors and articulate useful self-correction strategies.”
From Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most of them prompt or train language models to correct their initial results.”
From CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“various feedback and self-correction techniques have been introduced to improve LLM reliability, such as Reflexion and Self-Refine [21,22].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“Refinement -where solutions are instead critiqued and improved upon during resamplingoffers a possible avenue for breaking out of the aggregation rut.”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“MAGICORE also outperforms a combination of Self-Refine (Madaan et al., 2023) and Self-Consistency by 4.0%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-refinement methods (Madaan et al., 2023) show that models can improve through iterative critique or feedback, often using instructive signals or revision exemplars.”
From GIER: Gap-Driven Self-Refinement for Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“It has been demonstrated that prompting-only setups can work well for stylistic or qualitative improvements (e.g.Madaan et al., 2023;Chen et al., 2023b), but would require external feedback for reasoning tasks (Huang et al., 2023;Shinn et al., 2023;Kim et al., 2023;Madaan et al., 2023).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023), as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Meta-Harness is also closely related to methods such as ProTeGi, TextGrad, OPRO, GEPA, AlphaEvolve/OpenEvolve, and Feedback Descent, which iteratively improve prompts or other text artifacts using feedback from prior attempts [38; 31; 53; 51; 1; 35; 43; 26].”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“Building on early research showing that LLMs can produce useful feedback and iteratively improve outputs (Madaan et al., 2023), recent work has explored using multimodal models as visual critics to ground code generation and refinement.”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-Refine (Madaan et al., 2023) shows diminishing marginal returns across multiple rounds of self-refinement, with each additional iteration providing smaller gains.”
From Can LLMs Self-Correct Table Reasoning Errors? (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-Refine introduced iterative self-critique, demonstrating that models could improve their own outputs through multi-turn prompting loops [Madaan et al., 2023].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-correction methods enhance output quality through iterative refinement [6]”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“We observe only marginal improvements on REALINSTRUCT and a performance drop on IFEval using the Self-Refine (Madaan et al., 2023) pipeline, highlighting the limitations of traditional self-correction approaches for the multi-constrained instruction following task.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023)”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“and applying their self-correction prompt on top of model responses from our stronger version of the standard prompting again leads to a decrease in performance.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Table 8 demonstrates that our new prompt, denoted as Standard Prompting (ours) , significantly outperforms the results after self-correction of Madaan et al. (2023)”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“we take the Constrained Generation task in Madaan et al. (2023) as an example, where the task requires models to generate coherent sentences using all 20-30 input concepts.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Addressing the notable challenges of self-correction in large language models [Madaan et al., 2023, Huang et al., 2023b], AutoMix employs a non-LLM setup for routing and avoids escalating issues such as hallucination and reasoning errors [Dziri et al., 2023].”
From AutoMix: Automatically Mixing Language Models (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“First, 'self-improvement' is not one thing. The term is used for inference-time output revision (Self-Refine [6]), training loops on self-generated data (STaR [7], Self-Rewarding LMs [8]), agents that rewrite their own code (Gödel Agent [9], Darwin Gödel Machine [10]), and systems that autonomously conduct AI research [11]. These differ enormously in ambition and risk profile, yet the literature's”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“A key feature of the Code agent is its self-refinement mechanism [31], which enables iterative improvement of generated code.”
From SafeCodeRL: Security-Constrained Multi-Agent Reinforcement Learning for Trustworthy LLM-Generated IoT/CPS Software (2026) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“many recent studies have investigated the potential of LLMs to correct and refine their outputs, demonstrating that they indeed possess those capabilities (Welleck et al., 2023; Madaan et al., 2023; Shridhar et al., 2023; Ganguli et al., 2023; Wang et al., 2023b; Qi et al., 2023; Yang et al., 2023).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Similar to humans, LLMs do not always generate optimal outputs on their first attempt.”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Reflection-based methods, such as Reflexion [11] and Self-Refine [12], provide critical feedback that allows agents to learn from previous mistakes.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“Critique-and-refinement is a technique to improve the initial output of an LLM on a specific task.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies demonstrated that LLMs can serve as effective critics, generating detailed and constructive feedback [13, 14].”
From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Prompts used by Madaan et al. (2023) for generating initial responses instruct to generate responses that are not interesting and not very engaging, which are contradicting to the task goal.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, few existing approaches jointly evolve the full set of editable components [16]; most focus on a single component, typically the prompt [32, 50, 20], skills [19, 43], or an in-context playbook [49].”
From Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredIn the citing paper
“Few-shot examples for generating initial responses include examples in settings different from the target task (positive to negative), while all few-shot examples for refinement are positive to negative.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“However, most studies in intrinsic self-correction (Madaan et al., 2023;Huang et al., 2024a) use simple prompts for generating feedback, and there is room for improvement.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“As more obvious weak prompts, Self-Refine (Madaan et al., 2023) uses instructions or few-shot examples that do not correctly correspond to the target task only for initial response generation (e.g., providing wrong target labels in few-shot examples), while using appropriate instructions for self-correction, as shown in Table 9 and 10.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“For example, the constrained generation task evaluated in Self-Refine (Madaan et al., 2023) is a task to generate a sentence that includes five specified words.We can easily evaluate the correctness by checking whether the five words are included in the generated sentence.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room?
“| This work | SELF-REFINE w/ GPT-3.5 SELF-REFINE w/ ChatGPT | 62.4 75.1 | | | SELF-REFINE w/ GPT-4 | 94.5 |”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Math Reasoning 64.1 64.1 (0) 74.8 75.0 (↑0.2) 92.9 93.1 (↑0.2)”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“| SELF-REFINE wins | 36.0 | 48.0 | 54.0 | | INIT wins | 23.0 | 18.0 | 16.0 | | Both are equal | 41.0 | 50.0 | 30.0 |”
Reported 100 · recomputed 116
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“* Well-known: CSCLE is not a well-known acronym.5/5 * Total score: 23/25”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
Reported 23 · recomputed 24
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“We use greedy decoding with a temperature of 0.7 for all setups.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
“We consider the temperature of both T = 0.0 (greedy) and T = 0.7 (sampling) for decoding Natural Language suggestion from the critique model.We always use a temperature T = 0.0 (greedy) when decoding Programming Language from the code editor.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
“Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINEand ChatGPT+SELF-REFINE across all tasks”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Sentiment Reversal 8.8 30.4 (↑21.6)11.4 43.2 (↑31.8)3.8 36.2 (↑32.4)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“* Well-known: CSCE is not a well-known acronym.2/5 * Total score: 20/25”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
Reported 20 · recomputed 21
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Refine Feedback
“We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo;OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Dialogue Response 36.4 63.6 (↑27.2) 40.1 59.9 (↑19.8)25.4 74.6 (↑49.2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“The FEEDBACK-REFINE iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
“Due to budget constraints, we run SELF-REFINE for N = 5 iterations.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
“Table 13: SELF-REFINE results from table 1 with Wilson confidence interval (at 95% confidence interval) and statistical significance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
“Table 13 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at α= 99% confidence interval) and statistical significance.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2×
“We also experiment with Vicuna-13b (Chiang et al., 2023), a version of LLaMA-13b (Touvron et al., 2023) fine-tuned on conversations sourced from the web.”
BackgroundMethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481 · cited 11×
“Dataset: (Zhang et al., 2015) 1000 review passages x: The food was fantastic.”
BackgroundMethodInfluentialThe Architecture of Complexity
Simon, H. A. (1991) · Springer eBooks · doi:10.1007/978-1-4899-0718-9_31 · Cited by 4,479 · cited 1×
“Iterative self -refinement is a fundamental characteristic of human problem-solving (Simon, 1962; Flower and Hayes, 1981; Amabile, 1983).”
BackgroundChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Interval Estimation for a Binomial Proportion
Brown, L. D., Cai, T., & Dasgupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,746 · cited 1×
“Table 13 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at α= 99% confidence interval) and statistical significance.”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 4×
“5 (text-davinci-003 and gpt-3.5-turbo; OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
BackgroundInfluentialEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Research works printed but not linked: 3
- Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu. 2019. Teaching a black-box learner. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1547-1555. PMLR.
- William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022a. Self-critiquing models for assisting human evaluators.
- Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022a. Peer: A collaborative language model. · DOI · Paper page
Cited by
Sorted by citations
Items 301–400 of 778
Vulnerability of LLM Outputs to Heuristics-Inducing Prompt Structures
Kuramoto, T., Kamei, R., & Suzuki, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789108
iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision
Bai, J., Cheong, W. S., Müller, P., & Lim, B. Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790539 · Cited by 4
CReTIHC: Designing Causal Reasoning Tasks about Temporal Interventions and Hallucinated Confoundings
Chun, C., Lee, S., Seo, J., & Lim, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.693 · Cited by 2
“Inspired by the self-refinement method (Madaan et al., 2023), we integrated self-refining the sentences generated by LLM and rewriting inappropriate sentences into the data collection process.”
MethodFrom Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models
Yan, J., Wang, C., Zhang, T., He, X., Huang, J., & Zhang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.828 · Cited by 5
Mejía-Trejo, J. (2024) · doi:10.55965/abib.2024.9786076984505 · Cited by 3
Cost, Complexity, and Efficacy of Prompt Engineering Techniques for Large Language Models
Cherukuri, M. (2025) · International Journal on Science and Technology · doi:10.71097/ijsat.v16.i2.2584 · Cited by 1
Understanding De-dollarization Among BRICS Nations: Systematic Review of the Factors and Fallout
Zein, S., Clemente‐Almendros, J. A., & Boldeanu, F. T. (2025) · Revista de Administração Contemporânea · doi:10.1590/1982-7849rac2025240197.en · Cited by 6
ReproCopilot: LLM-Driven Failure Reproduction with Dynamic Refinement
Leesatapornwongsa, T., Faisal, F. E., & Nath, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729399 · Cited by 1
“Unlike Self-Refine [66], which simply asks the LLM to refine previously generated code, our approach uses dynamic analysis to identify program states and runtime conditions that the symptom and the code do not reveal but are necessary for reproduction.”
An end-to-end approach for fixing concurrency bugs via SHB-based context extractor
Li, Z., Yi, Q., Xiao, K., Ji, Z., & Liang, H. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112888 · Cited by 1
“Building on research that shows LLMs can improve their outputs or adapt based on feedback from external tools or human input—demonstrating the significant impact of these tools on enhancing model performance [42], [43]—we incorporate compiler or JPF error messages to refine the generated fixes.”
Multi-Agent LLM System for Automatic Meeting Information Processing
Varjú, Á., & Varga, P. (2025) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms57970.2025.11073611 · Cited by 1
Towards Robust Synthetic Data Generation for Simplification of Text in French
Tsourakis, N. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7030068 · Cited by 1
Shindo, N., & Uto, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99267-4_7 · Cited by 1
Trapped in the Prompt Loop: Reprompt Behavior in Writing with ChatGPT
Jung, H., Jeong, Y., & Lee, J. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2530100 · Cited by 1
“In this regard, there has been some research on the problem of iterative process in LLM prompting (Madaan et al., 2023).”
Pelayo-Benedet, T., Rodríguez, R. J., & Gañán, C. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104176 · Cited by 3
Wang, M., Li, Z., Zhao, X., & Guo, Q. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104422 · Cited by 1
Meta-reasoning in autonomous agents: performance gains across benchmarks and models
Talukdar, W., Biswas, A., Shankar, G., Shinde, V., & Parekh, G. (2026) · Academia AI and Applications · doi:10.20935/acadai8229 · Cited by 2
Kaplan, A., Kaplan, B. A., Aydilek, A., Ege, T., Şentürk, S., & Solmaz, İ. (2026) · European Spine Journal · doi:10.1007/s00586-026-10094-0 · Cited by 2
Ding, W., Chen, J., Chen, M., Xie, F., Wang, J., Yang, S. X., & Dames, P. (2026) · Frontiers in Robotics and AI · doi:10.3389/frobt.2026.1905578 · Cited by 2
PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering
Han, W., Park, J., & Lee, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.517 · Cited by 2
Wang, X., Li, L., & Zeng, D. D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.309 · Cited by 2
CMMQC: Cascaded Multi-Model Quality Control for Unsupervised Data-to-Text Generation
Zhang, W., Sun, X., Luo, Y., Gao, W., & Zhu, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650876 · Cited by 1
“For example, models can iteratively revise generated text using self-feedback [14].”
MethodToFC: Tree-of-Fact with Continued Best-First Search for Commonsense Reasoning
Dai, L., & Qin, K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649921 · Cited by 1
“Existent studies [29] [27] have pointed out that with specific prompts, language models can validate their own output.”
BackgroundMDE in the Era of Generative AI
Mdaghri, A. A., Ouederni, M., & Chaâri, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-85356-2_8 · Cited by 1
Zhao, K., Yang, B., Tang, C., Dai, S., Tang, H., Lin, C., & Zhan, L. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109574 · Cited by 2
Advance Toward Artificial Superintelligence with OpenAI's O1 Reinforcement Learning and Ethics
Mirindi, D., Sinkhonde, D., Mirindi, F., & Bezabih, T. (2025) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc64931.2025.11077494 · Cited by 1
Mathis, W. S., Zhao, S., Pratt, N., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4673015 · Cited by 1
40 questions for shaping a policy-salient Bitcoin research agendaPreprint
Rudd, M. A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4821335 · Cited by 1
Shen, S., Xiao, Z., & Han, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5382182 · Cited by 1
Exposing Hidden Bias: A Study of Fairness Debt in Gray LiteraturePreprint
Sotolani, R. S., Freire, S., Fronchetti, F., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5459109 · Cited by 1
Kuerbanjiang, W., Wang, X., Jiamaliding, Y., Maimaitiaili, M., & Yi, Y. (2025) · medRxiv · doi:10.1101/2025.10.30.25339199 · Cited by 1
Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
Chen, H.-Y. (2026) · arXiv · doi:10.48550/arxiv.2604.21611 · Cited by 1
SmartPlay: A Benchmark for LLMs as Intelligent Agents
Wu, Y., Tang, X., Mitchell, T. M., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.01557 · Cited by 6
“…models demonstrate exceptional planning (Ahn et al., 2022; Wu et al., 2023b), reasoning (Wu et al., 2023a; Shinn et al., 2023), and problem-solving (Madaan et al., 2023; Kim et al., 2023) abilities, enabling the potential as generalist agents for virtual-reality (Park et al., 2023) or real-world…”
BackgroundThe Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1
“As a training-free fix for our setting, we also try model-based self-verification (see Kamoi et al. (2024) for a survey) at each turn in Appendix C.1, which can be considered a form of self-refinement (Madaan et al., 2023) without having access to environment feedback (Gou et al., 2023).”
DarwinX: Evolving Agent Harnesses Through Natural Selection
Zhang, Y., Dai, Y., Tan, J., Yang, L., Mullur, R., Hoang, T., Hu, Z., Zhu, J., Mui, P., Savarese, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2608.07545 · Cited by 1
“Prompt optimizers search over instructions [12, 14, 49, 54, 60], and reflective variants reuse verbal feedback and trajectory distillation [2, 29, 34, 42], while DSPy and related programs compile prompts, demonstrations, or textual parameters against data [9, 23, 36].”
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Lin, J., Liu, S., Pan, C., Lin, L., Dou, S., Xi, Z., Huang, X., Yan, H., Han, Z., Gui, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25850 · Cited by 1
“Existing approaches mostly optimize individual harness components, typically the prompt [30, 44, 18] or an in-context playbook [43], with few jointly evolving the full set of editable components [15].”
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., Zheng, S., Choe, K. W., Sun, J., Wang, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12670 · Cited by 1
“…2023; Wang et al., 2023a), chain-of-thought and ReAct for multi-step problem solving (Wei et al., 2022; Yao et al., 2023; 2022; Shinn et al., 2023; Madaan et al., 2023; Zhou et al., 2023; 2024a), and retrieval/tool use (Lewis et al., 2020; Zhou et al., 2022; Schick et al., 2023; Qin et al.,…”
FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
Dai, D., Liu, M., Li, A., Cao, J., Wang, Y., Wang, C., Peng, X., & Zheng, Z. (2025) · arXiv · doi:10.48550/arxiv.2504.06939 · Cited by 1
“With the rise of large language models (LLMs), there is increasing interest in leveraging their capabilities for automated code repair by incorporating test feedback and compiler diagnostics [30, 34].”
Robeyns, M., Szummer, M., & Aitchison, L. (2025) · arXiv · doi:10.48550/arxiv.2504.15228 · Cited by 117
A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue
Pałgan, B., Wawryszczuk, M., Chwał, J., Tkacz, E., & Dzik, R. (2026) · Applied Sciences · doi:10.3390/app16094329 · Cited by 1
EvoX: Meta-Evolution for Automated Discovery
Liu, S., Agarwal, S., Maheswaran, M., Cemri, M., Li, Z., Mang, Q., Naren, A., Boneh, E., Cheng, A., Pan, M. Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.23413 · Cited by 1
Harness^2: Recursive Agent Harnessing for an Open World
Xu, R., Chen, Y., CuiZhu, Z., Mishra, B. D., Ming, Y., Yu, H., Han, R., Lee, C.-Y., & Pfister, T. (2026) · arXiv · doi:10.48550/arxiv.2610.14224 · Cited by 1
“Early approaches refine outputs through self-feedback or retain reflections for subsequent attempts (Madaan et al., 2023; Shinn et al., 2023).”
TT-SI: Self-Improving LLM Agents with Test-Time Training
Acikgoz, E. C., Qian, C., Ji, H., Hakkani-Tür, D., & Tur, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.462 · Cited by 1
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
Dycke, N., Zečević, M., Kuznetsov, I., Suess, B., Kersting, K., & Gurevych, I. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1107 · Cited by 1
Dycke, N., & Gurevych, I. (2025) · arXiv · doi:10.48550/arxiv.2508.21422 · Cited by 1
Do as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1
“Llama3-70B Qwen2-72B Another commonly used strategy to enhance LLM reasoning, apart from modifying system prompts, is introducing of a reflection mechanism [55–57].”
Vladika, J., Domres, A., Nguyen, M., Moser, R., Nano, J., Busch, F., Adams, L. C., Bressem, K. K., Bernhardt, D., Combs, S. E., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.24830 · Cited by 1
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Liu, X., Lan, B., Hu, Z., Liu, Y., Zhang, Z., Wang, F., Shieh, M., & Zhou, W. (2024) · arXiv · doi:10.48550/arxiv.2408.03910 · Cited by 1
ALOBED, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7291548
Self-enhancing prompt optimization for language style generation
Liu, H., He, H., Yang, W., Zhou, J., & Zhao, N. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116519
Switch, Reason, and Revise: Enhancing Reasoning Capability of Video Game AI by Large Language Models
Li, W., Liu, H., Lv, J., Huang, K., Song, A., & Lei, Z. (2025) · IEEE Transactions on Games · doi:10.1109/tg.2025.3630322
VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection
Wang, Q., Gao, X., He, Y., Han, J., Li, J., Dong, S., Ma, Z., & Gong, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37592-6_30
Lim, S.-Y., Sato, K., Takami, K., Parady, G., & KIM, E. (2026) · Travel Behaviour and Society · doi:10.1016/j.tbs.2026.101374 · Cited by 1
Bao, M., Zhang, K., Liu, X., Liu, Q., Zhao, H., & Chen, E. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3658068
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135630
Architectural Laws And Derived Constraints Of Verification And Consequential AI ReliancePreprint
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135727
A Consistency-Oriented Verification Framework for Reliable Power Industry LLMs
Wang, B., Zhang, L., Li, Y., & Guo, N. (2025) · International Conference on Electronic Information Engineering and Computer Communication (EIECC) · doi:10.1109/eiecc67963.2025.11409479
Cognitive Consistency Verification: Neuro-Symbolic Approach for Mitigating VLM Hallucination
Jo, H. S., & Kwon, Y. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402484
Hu, X., Zhang, Q., Miao, J., & Wang, P. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401656
Jang, P.-S., & Lee, J.-Y. (2026) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2026.24.2.169
MATCH: Multi-Agentic Evidence Grounding for Explainable Hate Video Detection
Li, K., Hong, R., Lang, J., Wu, J., Zhou, F., & Song, J. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3672052 · Cited by 2
Iterative Chain-of-Thought: Self-Correcting Reasoning via Verifier-Guided Feedback Loop
Thanjavuru, T. (2026) · Proceedings, annual Reliability and Maintainability Symposium/Proceedings. Annual Reliability and Maintainability Symposium · doi:10.1109/rams50514.2026.11424546
Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systems
Hosseini, M. P., Shah, A., Qureshi, S., Huang, A., Miao, C., & Wei, W. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00034 · Cited by 1
Yenikaya, M. A., & Odabaşoğlu, M. L. (2026) · 3 SEKTÖR SOSYAL EKONOMİ DERGİSİ · doi:10.63556/tisej.2026.1731
An Agentic AI Framework for Strengthening Preventive Healthcare
Arul Kumaran, G., Balasastha, E., Jayakumar, N., & Jayakumar, R. (2026) · International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS) · doi:10.1109/iciscois62701.2026.11447737
Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey
Zhao, Y., Liu, Z., Zheng, Y., & Lam, K. (2026) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic68212.2026.11454197 · Cited by 4
Prompt Optimization Framework for Enhancing Large Language Model Response Accuracy
Sukanya, N. S., Asha, V., Srujan, D., Lasrado, S., Patil, S., & Kalgutkar, S. D. (2026) · International Conference on Sentiment Analysis and Deep Learning (ICSADL) · doi:10.1109/icsadl67539.2026.11451930
Low-Cost Verification for Multimodal Large Language Models
Zhang, S., Lü, G., & Ma, Z. (2026) · International Conference on Communication Networks and Machine Learning (CNML) · doi:10.1109/cnml68938.2026.11453286
Large Language Model-Based Smart Agents for Attack-Defense Game on Cyber-Physical Power Systems
Wang, B., Zhou, Y., Wang, S., Xu, B., & Deng, R. (2026) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2026.3681079 · Cited by 2
Diverse and efficient ensembling of deep networks
Ramé, A. (2023) · theses.fr (ABES) · doi:10.70675/fa295dcfz969ez4448z9a30z7c7bbfabb740
Data Analysis on FHIR through Visual Tools and Generative AI
Damiano, E., & Orciuoli, F. (2026) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa67686.2026.11468114
Rudra, A., Karumuri, M. S., & Agrawal, M. (2026) · Proceedings of IEEE Southeastcon · doi:10.1109/southeastcon63549.2026.11476187
LightOL: A Lightweight Ontology Learning Framework with Large Language Models
Liu, Y., Liu, B. F., Dong, B., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464515
Zhao, M., Tianxi, H., Duan, G., & Wang, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463795
Reflective Confidence: Correcting Reasoning Flaws via Online Self-Correction
Zeng, Q., Yang, J., & Wang, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463123
Enhancing Value Alignment of LLMs with Multi-Agent System and Combinatorial Fusion
Wu, Y., Bouneffouf, D., & Hsu, D. F. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460611 · Cited by 1
Breaking Cognitive Fixation in Multi-Turn Dialogue with Self-Distancing and Incubation
Gwon, H.-M., Huang, J., & Lee, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464620
Nguyen, H., Zhang, H., Zhang, Y., Ding, J., & Chen, H. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00042 · Cited by 1
LLM-ReSum: A Framework for LLM Reflective Summarization Through Self-Evaluation
Nguyen, H., Zhang, H., Zhang, Y., Chen, H., & Ding, J. (2026) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice70481.2026.00012
Zero-Shot Composed Image Retrieval via Dual-Stream Instruction-Aware Distillation
Zhong, W., Barton, R., An, W., Jiang, F., Ma, H., Guo, Y., Dan, A., Sam, S., Bouyarmane, K., & Huang, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02063
TARA-LLM: A Knowledge-Driven Model for Task Assignment of Multiplayer Reach-Avoid Games
Liang, R., Yan, R., Liu, B., Li, X., Chen, J., & Dong, X. (2026) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2026.3683214
Ghosal, R., Middya, A. I., & Roy, S. (2026) · IEEE International Conference on AI Engineering and Innovations (AIEI) · doi:10.1109/aiei69164.2026.11496917
Meem, U. S., Saif, S. B., Azad, N. B., Hossain, N., Kashem, S. B., & Ifty, R. A. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11490270
CoV-ME: Memory Editing for Self-Consistent Reasoning in Large Language Models
Hossain, M. T., Nourani, M. S., & Sultana, N. (2025) · IEEE International Conference on Robotics, Automation, Artificial-Intelligence and Internet-of-Things (RAAICON) · doi:10.1109/raaicon69033.2025.11502550
A systematic review of meta-cognitive approaches to AGI
Nishit Bohra, M., Wagle, S. A., & Patil, S. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01401-1
Exp-ToolPlan: Contrastive Experience Learning for Similar Tool Selection
Cheng, Y., Yan, M., & Wan, Z. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519215
STAR: A Stage-Attributed Triage and Repair Framework for RCA Agents in Microservices
Wang, J., Chen, R., Wang, Y., Liao, X., Fan, Z., Wu, X., Wu, W., Zhao, J., & Liu, X. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00057 · Cited by 2
Tarannum, A., Mohammed, M. A., Mandalawi, S. A., Cakmak, M. C., & Talburt, J. R. (2026) · Knowledge · doi:10.3390/knowledge6030023
ProxyLLM: Augmenting LLMs With Proxy Models for Tool Utilization in Network Service Generation
Zhou, X., Shao, Z., Jia, Q., & Xie, R. (2026) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2026.3695074
Controllable Visions: Enhancing Text-to-Image Generation via Standardized Prompt Rewriting
Meng, F., Xing, J., Guo, H., Zeng, Z., Guan, Y., & Lei, B. (2025) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait67930.2025.11522610
AgentEvo: cost-aware agentic workflow generation via adaptive multi-stage evolution
Xiao, B., Liu, H., Wang, Y., Hou, Y., Xu, K., & Zhang, X. (2026) · Complex & Intelligent Systems · doi:10.1007/s40747-026-02325-0
A Multi-Agent Cognitive Isolation Framework for Defect-Driven Power Grid Document Assessment
Yang, J., Ao, Z., Ren, H., Zhu, H., Ding, L., Wang, M., Jiang, X., & Yang, Q. (2026) · Asia Conference on Power and Electrical Engineering (ACPEE) · doi:10.1109/acpee69242.2026.11524710
Koc, V. (2026) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds69358.2026.11540303
(15128), J. L., Zhang, Y., Ning, T., Zhao, S., & Teng, Y. (2026) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp69961.2026.11540650
Diamond of Thoughts: Synergistic Merging for Efficient Large Language Model Reasoning
Zhang, Y. (2026) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp69961.2026.11540838
Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation
Khan, M. Z. S., Azar, K. Z., & Kamali, H. M. (2026) · doi:10.23919/date69613.2026.11539716
Towards Agentic AI for Event Log Data Quality Assessment
Comuzzi, M., Ra, S., Narmanova, D., Cho, S., Hong, Y., & Jung, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28117-3_23
Comparative Evaluation of Prompting Strategies for LLM-Based Combinatorial Optimization
Chadli, T., & Mourhir, A. (2026) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset68627.2026.11538595
Tang, Y., Liu, Z., Chen, Z., Yang, J., Xiang, L., Xu, Z., & He, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9875246/v1
Metadata
- Authors
- Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 46534-46594
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.17651
- OpenAlex
- W7133236450
- Cited by
- 2,314
- References
- 52
- Citations by year
- 2026: 1329 · 2025: 460 · 2024: 338 · 2023: 186
- FWCI
- 260.57
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- iterative refinement, self-feedback, large language models, mathematical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
