The DOI matches its DataCite record; 47 of the 53 references it cites were found in a registry or an index; 12 of the 22 sources its claims cite were checked for support.
Self-Refine: Iterative Refinement with Self-Feedback
Open accessAbstract
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
SELF-REFINE improves large language model outputs by iteratively generating self-feedback and refining the same model’s output without supervised training, additional training, or reinforcement learning. Across 7 diverse tasks, it improves over one-step generation by ∼20% absolute on average, with especially large gains on preference-based tasks.
Key findings
- SELF-REFINE consistently improved GPT-3.5, ChatGPT, and GPT-4 across all evaluated tasks. Examples include GPT-4 Dialogue Response Generation improving from 25.4 to 74.6 (+49.2), GPT-4 Code Optimization from 27.3 to 36.0 (+8.7), and GPT-4 Constrained Generation from 15.0 to 45.0 (+30.0).
- Feedback quality matters: in Code Optimization, SELF-REFINE feedback scored 27.5 vs. 26.0 for generic feedback and 24.8 for no feedback; in Sentiment Reversal, 43.2 vs. 31.2 vs. 0; in Acronym Generation, 56.4 vs. 54.0 vs. 48.0.
- Iterative refinement generally improves outputs, with diminishing returns: Code Optimization rose from 22.0 at y0 to 28.8 at y3; Sentiment Reversal from 33.9 to 36.8; Constrained Generation from 29.0 to 49.7.
- Math Reasoning gains were modest: GPT-3.5 64.1 to 64.1 (0), ChatGPT 74.8 to 75.0 (+0.2), GPT-4 92.9 to 93.1 (+0.2). With oracle feedback, GPT-3 improved by 4.8% and GPT-4 by 0.7%.
Methods
- SELF-REFINE alternates between FEEDBACK and REFINE using the same model M: initial output y0 = M(pgen ∥ x); feedback fbt = M(pfb ∥ x ∥ yt); refinement yt+1 = M(prefine ∥ x ∥ y0 ∥ fb0 ∥ … ∥ yt ∥ fbt). It iterates until a stopping condition or up to a maximum of 4 iterations, using few-shot prompts and no supervised training, additional training, or reinforcement learning.
- Evaluation covered 7 tasks: Dialogue Response Generation, Code Optimization, Code Readability Improvement, Math Reasoning, Sentiment Reversal, Acronym Generation, and Constrained Generation (CommonGen-Hard). Base LLMs were GPT-3.5 (text-davinci-003), ChatGPT (gpt-3.5-turbo), GPT-4, and CODEX for code tasks. Metrics included task-specific metrics, human preference, and GPT-4 preference.
- Analysis included ablations on feedback quality and iteration count, comparison against generating multiple outputs, testing Vicuna-13B, and a qualitative error analysis of 70 samples for Code Optimization and Math Reasoning.
Limitations
- Stated: the base models need sufficient few-shot modeling or instruction-following abilities to provide feedback and refine in-context; Vicuna-13B struggled to generate feedback in the required format and often failed to refine properly.
- Stated: experiments used non-open-sourced models (GPT-3.5, ChatGPT, GPT-4, CODEX), whose pretraining corpus, sizes, and biases are not fully described, and which are not free to use; datasets were exclusively English; the approach does not explicitly guard against bad actors steering models toward toxic or harmful text.
- Evident from the text: human A/B evaluations were conducted by the authors, and GPT-4 was used as a proxy for human preference with reported correlations of 82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Generation. The paper also notes Math Reasoning gains are limited by the model’s inability to accurately identify errors, with ChatGPT feedback being “everything looks good” for 94% of instances.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
70 Own result · 25 Own interpretation · 9 No basis given · 7 Partly supported · 5 Own method · 4 Supported · 4 Sources not found · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“In code-generation tasks, SELF-REFINE improves the initial generation by up to absolute 13% when applied to strong code models such as Codex (code-davinci-002; Chen et al., 2021).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Refine Feedback
What the sources say
- Supportedcites Chen et al., 2021 source
The Chen et al. citation sits in the parenthetical identifying Codex, rather than after the reported SELF-REFINE improvement. Section 3, “Code Fine-Tuning” (p. 4), describes producing Codex and evaluating it on code-generation tasks, supporting that identification. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, these reinforcement learning methods require updating the model's parameters, unlike SELF-REFINE.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Supportedcites Stiennon et al., 2020 source
The Introduction (p. 2) says the policy is updated using PPO, and §3.4 (p. 6) describes a learned RL policy with parameters. This supports the claim about the reinforcement-learning methods discussed; the contrast with SELF-REFINE is the citing authors’ framing, not a finding attributed to this source. (LLM, conf=0.90)
- Not matchedcites Lu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Stiennon et al. (2020) source
The Introduction (p. 2) says the policy is trained through reinforcement learning and updated using PPO. Section 3.4 (p. 6) also describes the learned RL policy as having parameters. The comparison with SELF-REFINE is the citing authors’ own framing, not a claim this source is needed to establish. (LLM, conf=0.92)
- Not matchedcites Lu et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al. (2022a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This adjustment notably enhanced performance in the Math Reasoning task, with GPT-3 improving by 4.8% and GPT-4 by 0.7% Table 9.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
What the sources say
- Partly supportedcites Welleck et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.5 (pp. 8–9) reports that correcting only incorrect outputs and adding explicit feedback improve results in math program synthesis. It does not report the claimed 4.8% GPT-3 or 0.7% GPT-4 gains; its math results use different models and figures. The cited paper’s Table 9 concerns E2E hyperparameters, not those gains. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These methods differ from SELF-REFINE (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Partly supportedcites Stiennon et al., 2020 sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says this paper’s RL policy is updated using a reward for the entire generated summary, and a note on p. 6 explicitly says the reward model gives no rewards at intermediate time steps. That supports the distinction for this method, but not the broader claim that reinforcement-learning approaches generally cannot access feedback on intermediate generations. (LLM, conf=0.84)
- Not matchedcites Lu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Le et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked to generate a sentence containing up to 30 given concepts.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 4 demonstrates that on average, the quality of the output improves as the number of iterations increases.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all evaluated tasks, outputs generated with SELF-REFINE are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ∼20% absolute on average in task performance.”
- Also stated in: § 3.3 Results · plain
“SELF-REFINE consistently improves over base models across all model sizes, and additionally outperforms the previous state-of-the-art across all tasks.”
- Also stated in: § Refine Feedback · plain
“We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo;OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.3 Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, SELF-REFINE leads to especially high gains.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in the Code Optimization task, the initial output (y 0 ) has a score of 22.0, which improves to 28.8 after three iterations (y 3 ).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 35
- The quote only partly states this claim. 5
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“CommonGen-Hard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only the 3-5 related concepts given in CommonGen.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > K New Tasks
What the sources say
- Partly supportedcites Lin et al., 2020 sourceExperimental — accuracy not yet measured
Section 3.1 says CommonGen uses concept sets of three to five concepts, and Section 2 says the output should be a grammatical sentence using all given concepts. Our copy does not mention CommonGen-Hard or a requirement to incorporate 20–30 concepts (searched for “CommonGen-Hard” and “20-30 concepts”), so that part is unsupported. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing literature (Ouyang et al., 2022) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
What the sources say
- Partly supportedcites Ouyang et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.5 (page unavailable) calls the pretrained models’ behavior “poorly characterized,” which lends some support to the concern about incomplete characterization. But the paper does not say that existing literature omits the listed details; its Introduction gives the model sizes, and Section 5.3 discusses biased outputs. The claimed lack of disclosure about model sizes and biases is therefore not supported. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirically, we evaluated SELF-REFINE using the same base model of GPT-3 as Self-Correction, and with the same settings on the GSM8K benchmark.Self-Correction achieved 45.9% accuracy while SELF-REFINE (this work) achieved 55.7% (↑9.8).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Partly supportedcites Welleck et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.4 (p. 8) reports 45.92% accuracy on GSM with a GPT-3 Instruct generator and self-corrector, supporting the rounded 45.9% figure. The paper does not report SELF-REFINE’s 55.7% result or establish that the two methods used the same settings, so it does not substantiate the 9.8-point comparison. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Self-Correction trains a separate refiner (or”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Supportedcites Welleck et al., 2022 source
Section 2 (p. 2) says Self-Correction keeps the generator separate and trains the corrector with different objectives for different task requirements. The Introduction (pp. 1–2) also describes a trained corrector and its use across three tasks. The comparison describing SELF-REFINE’s prompting approach is not a claim about this source. (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Few-shot prompting (also referred to as”
What the sources say
- Supportedcites Brown et al., 2020 source
Section 2 Approach (p. 6) defines few-shot as giving the model K task examples as conditioning. It describes each example as a context and desired completion—the input-output pairs in the claim—followed by a final context for the model to complete. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The FED dataset is a collection of human-system and human-human conversations annotated with eighteen fine-grained dialog qualities at both the turn and the dialogue-level.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
What the sources say
- Supportedcites Mehri and Eskenazi, 2020 source
The Introduction describes the FED dataset as human quality annotations of human-system and human-human conversations. Section 3 (Data Collection) specifies eighteen fine-grained qualities: eight annotated at the turn level and ten at the dialogue level. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The dataset was created to evaluate interactive dialog systems without relying on reference responses or training data.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
What the sources say
- Partly supportedcites Mehri and Eskenazi, 2020 sourceExperimental — accuracy not yet measured
The Introduction says the FED dataset was collected for fine-grained evaluation of interactive dialog and contains no training data. It also says the dataset is intended to benchmark automatic metrics against human judgments; the absence of reference responses is a property of the FED metric, not a stated purpose or property of the dataset. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This effect is more pronounced in tasks like Sentiment Transfer, where changing from our feedback to generic feedback leads to a significant performance drop (43.2 to 31.2), and the task fails without feedback.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Code Optimization, performance slightly dips from 27.5 (SELF-REFINE feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINEand ChatGPT+SELF-REFINE across all tasks, even in tasks where the initial base results of GPT-4 were lower than GPT-3.5 or ChatGPT.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To address this, SELF-REFINE's feedback generates explicit numerical scores to capture the different aspects of output quality.This allows for a more balanced evaluation of outputs and the selection of the most appropriate one.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Vicuna-13b was able to consistently follow the task initialization prompt.However, it struggled to follow the prompts intended for feedback and refinement.This often led to outputs that resembled assistant-like responses, a representative example of which can be found in Appendix G.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > G Evaluation of Vicuna-13b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 12 breaks down errors found in the refinement stage of SELF-REFINE.Errors include: not being robust (10%), ignoring feedback (25%), and introducing a new problem (20%).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback and Refinement Error Analysis for Response Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that nearly all of GPT-4 gains are statistically significant, ChatGPT gains are significant for 4 out of 7 datasets, and GPT-3.5 gains are significant for 3 out of 7 datasets.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: GPT-4 is capable of assessing and providing reasoning for complex tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE demonstrates its iterative feedback and refinement capabilities in the context of website layout generation.ChatGPT initially produces a rudimentary layout for a given topic, and then uses the FEEDBACK to suggest specific, actionable improvements, as demonstrated in Figures 7 and 9.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > I Beyond Benchmarks
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Starting from a rudimentary initial design, SELF-REFINE refines HTML, CSS, and JS to evolve the website in terms of both usability and aesthetics.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Going Beyond Benchmarks
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the Dialogue Response Generation task, SELF-REFINE can ignore good feedback but in a majority of cases, it is robust to bad feedback and ignores bad feedback.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, SELF-REFINE achieves superior performance while using only 4 samples at most, significantly fewer than the 16 and 32 samples employed by other models.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > F Comparison of SELF-REFINE with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance may not always monotonically increase with iterations: in multi-aspect feedback tasks like Acronym Generation, where the output quality can vary during iteration with improvement in one aspect but decline in another aspect.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 11 reports the occurrence of different types of errors in our sample, which includes Incorrect Feedback (25%), Generic Feedback (30%), and Incorrect Scoring (10%).”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback and Refinement Error Analysis for Response Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A similar selection is possible for other tasks like Math Reasoning and Sentiment Reversal, while we observe that output quality increases monotonically with iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > H.1 Using Oracle Feedback
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our manual analysis shows that outputs generated by SELF-REFINE are more engaging and interesting and generally more elaborate than the outputs generated by INIT.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE w/ GPT-3 | 55.7”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“CommonGen-Hard requires models to generate coherent and grammatically correct sentences incorporating 20-30 concepts, as opposed to the original task which presents a set of 3-5 related concepts.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE wins | 36.0 | 48.0 | 54.0”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“INIT wins | 23.0 | 18.0 | 16.0”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE w/ GPT-4 | 94.5”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“PaL w/ GPT-4 | 93.3”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE -feedback | 1 | 10.1 | 62.15 | 3.03”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“SELF-REFINE | 1 | 15.3 | 59.64 | 2.90”
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the FEEDBACK step of SELF-REFINE.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For dramatic response evaluation, we found that the preference rate went down drastically to 58.92%, from 80.09%.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > O Math Reasoning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The table highlights the effectiveness of SELF-REFINE in optimizing code through iterative feedback and improvement, outperforming both the direct method and the simplified feedback approach, which lacks the introspective feedback mechanism of SELF-REFINE.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Direct | - | 9.7 | 62.29 | 3.09”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“SELF-REFINE | 2 | 15.6 | 65.60 | 3.74”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“PaL w/ ChatGPT | 74.2”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sentiment Transfer | 75.00 | 21.43 | 3.57”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“Acronym Generation | 44.59 | 12.16 | 43.24”
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
“Response Generation | 47.58 | 19.66 | 32.76”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Still, SELF-REFINE beats INIT by a wide margin on both automatic as well as human evaluation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > M.2 Setup and Experiments
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At T = 0.0, SELF-REFINE produces less meaningful variables, less comments per line but even more function units.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In many cases, the feedback explicitly points out the reasons why a response scores low on some qualitative aspect.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Feedback
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At T = 0.7, SELF-REFINE produces more meaning variables, more function units and slightly more comments compared to the human annotators on average.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, this task has an extremely large number of reasonable outputs, and thus SELF-REFINE allows to better explore the space of possible outputs.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe that this task benefits significantly from SELF-REFINE because there are more opportunities to miss some of the concepts on the first attempt, and thus SELF-REFINE allows the model to fix these mistakes subsequently.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, the experiments in this work were performed with language models that are not open-sourced, namely GPT-3.5, ChatGPT, GPT-4, and CODEX.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Another limitation of our work is that we exclusively experiment with datasets in English.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SELF-REFINE focuses on iterative creation with introspective feedback, making it suitable for evaluating the effectiveness of language models on the CommonGen-Hard task.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > K New Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This demonstrates the potential of SELF-REFINE in real-world, complex, and creative tasks.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Going Beyond Benchmarks
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Compared to a concurrent work, Reflexion (Shinn et al., 2023), our approach involves correction using feedback, whereas their setup involves finding the next best solution in planning using ReAct.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
What the sources say
- Not matchedcites Shinn et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the limited performance of Vicuna-13b suggests that this model may require more extensive prompt-engineering for optimal performance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > G Evaluation of Vicuna-13b
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Nonetheless, we release our code and model outputs to ensure the reproducibility of our work.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other languages, the current models may not provide the same benefits.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 6 Limitations and Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, having multiple FEEDBACK-REFINE iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This distinction allows our method to offer more precise and actionable feedback, making it suitable for a wider range of natural language generation tasks, including those that may not necessarily involve step-by-step planning such as open-ended dialogue generation.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > B Broader Related Work
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We thus hypothesize that since Vicuna-13B was trained on conversations, it does not generalize as well as instruction-based models to test-time few-shot tasks.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 4 Analysis
What the sources say
- Not matchedcites Chiang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirectSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The setup was blind, i.e., the judges did not know which outputs were generated by which method.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > C Human Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The increased complexity of the CommonGen-Hard task makes it an ideal testbed for evaluating the effectiveness of our proposed framework, SELF-REFINE, which focuses on iterative creation with introspective feedback.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The judges were not aware of the method that generated each sample.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, acronym generation is a natural method testbed for our approach.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Acronym generation requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.
- Rests on an assumption: Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.
In the paper
“The primary objective of PIE is to optimize a given program by implementing algorithmic modifications that lead to improved runtime performance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
What the sources say
- Not matchedcites Madaan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through iterative creation and self-reflection, the SELF-REFINE framework empowers language models to progressively enhance the quality of their output, closely mimicking the human creative process and demonstrating its ability to improve generated text on complex and demanding natural language generation tasks like CommonGen-Hard (Figure 15).”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility.
In the paper
“Performance-Improving Code Edits or PIE (Madaan et al., 2023) focuses on enhancing the efficiency of functionally correct programs.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > N Code Optimization
What the sources say
- Not matchedcites Madaan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This significant increase in the number of concepts tests the model's ability to perform advanced commonsense reasoning, contextual understanding, and creative problem-solving, as it must generate meaningful sentences that encompass a broader range of ideas.”
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room? > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper relies on this without support.
Large language models often fall short in addressing intricate requirements.
“Although large language models (LLMs) can generate coherent outputs, they often fall short in addressing intricate requirements.”
The paper relies on this without support.
Math errors can be nuanced and sometimes limited to a single line or incorrect operation.
“In math, errors can be nuanced and sometimes limited to a single line or incorrect operation.”
The paper relies on this without support.
GPT-4 is capable of assessing and providing reasoning for complex tasks.
“In light of the impressive achievements of GPT-4 in assessing and providing reasoning for complex tasks, we leverage its abilities for evaluation in SELF-REFINE.”
The paper relies on this without support.
Code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase.
“though not related to the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase.”
The paper relies on this without support.
Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance.
“Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach.”
The paper relies on this without support.
Automated metrics may not provide an accurate assessment of text generation tasks.
“However, we acknowledge that automated metrics may not provide an accurate assessment of text generation tasks and rely on human evaluation instead.”
The paper relies on this without support.
Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics.
“Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics.”
The paper relies on this without support.
text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts, making GPT-3.5 a strong baseline.
“text-davinci-003 is capable of generating human-like responses of great quality for a wide range of dialogue contexts and hence GPT-3.5 is a strong baseline.”
The paper relies on this without support.
Acronym generation requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.
“Like in email writing, acronym generation also requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase.”
The paper relies on this without support.
Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.
“Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase.”
The paper relies on this without support.
Some acronym quality criteria are difficult to quantify and are a matter of human preference.
“Some of these criteria are difficult to quantify, and are a matter of human preference.”
The paper relies on this without support.
Modern LLMs have superior instruction following capabilities.
“As with other modules, we leverage the superior instruction following capabilities of modern LLMs to instead provide a few demonstrations of each task.”
The paper relies on this without support.
Initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility.
“Given that initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Self-Refine is effective but is limited to single-generation reasoning tasks.”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [15]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, this approach is suboptimal as it lacks problem-specific modeling, that is it does not recognize the constraints to be followed, and only prompts the model to evaluate and improve its responses without clear guidance on what to focus on.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al. (2023)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Retry-based refinement methods Madaan et al. [2023], Gou et al. [2024] often discard the entire context and regenerate from scratch, losing correct intermediate work.”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Madaan et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“They unfairly instruct the models to generate initial responses that have problems intentionally, over-evaluating selfcorrection performance.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research has shifted toward enabling agents to autonomously self-improve when provided with suitable interactive environments.”
From A Survey of Large Language Models (2026) · cites this paper as [277]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We evaluate FACTOOL against two baselines that use LLMs to check their own inputs: Self-Check with 3-shot CoT and zero-shot CoT, which have shown to been effective on various tasks including dialogue response, math reasoning, and code generation (Madaan et al., 2023;Chen et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The original prompt in Madaan et al. (2023) (Figure 7) does not clearly specify that the LLM needs to include all concepts in the prompt”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Yet a sobering empirical pattern has emerged: most autonomous self-improvement loops exhibit rapid asymptotic saturation. For instance, the Self-Refine paradigm plateaus after a mere few iterations [7].”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“while self-refinement(Madaan et al., 2023) and verification loops (Tang et al., 2023) iteratively critique and revise generations, improving factuality at the cost of higher inference overhead and a risk of over-editing.”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, unlike approaches such as Refine (Madaan et al., 2023), which struggle when combined with existing models on complex tasks”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some of the benchmarks (e.g. HotPotQA [18], GSM-8k [29], ) used in evaluations of self-verification are easy-that is, SoTA LLM performance is already high-and are therefore much less informative about the effects of the refinement procedure.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This belief seemingly rests on the assumption that verification of correctness should be easier than generation-a rather classical argument from computational complexity”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The self-refine approach was configured with four iterative refinement rounds at a temperature of 0.7, as adopted in prior research22.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Failed attempts and suboptimal decisions may accumulate, while new evidence from testing can invalidate earlier assumptions [5, 24, 33].”
From Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“existing text optimizers are poorly matched to this setting because they compress feedback too aggressively: they are memoryless, condition only on scalar scores, or restrict feedback to short templates or summaries.”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The reflection process is designed to enhance the system's capability to learn from mistakes within trajectories rather than only successes (Madaan et al., 2023; Shinn et al., 2023).”
From R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior self-reflection methods [11, 12] generate critiques of the LLM itself based on previous trial trajectories, leveraging past mistakes to guide the next trial.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While Self-Refine uses the model itself as its critic without additional context, our DECRIM pipeline employs a critic with fine-grained evaluation, assessing each constraint individually.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“But many works in this direction, such as Branch-SolveMerge (Saha et al., 2024) and Self-Refine (Madaan et al., 2023), assume constraint independence or focus on specific constraint types, limiting their applicability to real-world use cases.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“prior works have demonstrated the successful usage of self-correction that aligns model responses with specific preferences, such as altering the style of responses or enhancing their safety (Bai et al., 2022; Ganguli et al., 2023; Madaan et al., 2023).”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, systems like Voyager (Wang et al., 2023a), Self-Refine (Madaan et al., 2023), and SwarmAgentic (Zhang et al., 2025) employ profile-like adaptation and iterative self-improvement, but remain task-specific.”
From SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoning (2025) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some frameworks already experiment with using feedback from assertions or user grades to refine prompts [35, 39, 48].”
From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023;Chen et al., 2023c;Lu et al., 2023;Kim et al., 2023).”
From Large language models as tool makers (2023) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The generate-test-repair loop is Reflexion-style iteration [Shinn et al., 2023, Madaan et al., 2023, Chen et al., 2024], and the symbolic analogue of learning an action model inside a planning loop is classical action-model learning [Lamanna et al., 2021, Yang et al., 2007, Aineto et al., 2019].”
From OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3 (2026) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“although self-correction research mostly focuses on reasoning tasks such as math reasoning and coding (Madaan et al., 2023;Gou et al., 2024;Huang et al., 2024a).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The few-shot examples used by Madaan et al. (2023) for generating initial responses unfairly have different properties from the target task.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In specific, [62] proposes a self-refine mechanism.This mechanism consists of three crucial components: output, feedback, and refinement.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“This technique is increasingly crucial for emerging multicall LLM workloads such as Tree-of-Thoughts [52], Skeletonof-Thought [30], MCTS [55], and Self-Refine [26], where there are substantial opportunities for prefix sharing.”
From Locality-aware Fair Scheduling in LLM Serving (2025) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“thus, they show that self-correction improves task performance by asking the model to identify missing concepts and then guiding it to incorporate these concepts through feedback.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Self-reflection and iterative refinement methods (Madaan et al., 2023; Mousavi et al., 2023; Han et al., 2025c) naturally align with short-term, inside-trial memory, as the agent repeatedly leverages intermediate reasoning traces or outcomes from prior attempts within the same task.”
From Memory in the Age of AI Agents (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While effective, these approaches add an additional layer of complexity, requiring the model to both recognize its own errors and articulate useful self-correction strategies.”
From Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Most of them prompt or train language models to correct their initial results.”
From CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“various feedback and self-correction techniques have been introduced to improve LLM reliability, such as Reflexion and Self-Refine [21,22].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“Refinement -where solutions are instead critiqued and improved upon during resamplingoffers a possible avenue for breaking out of the aggregation rut.”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“MAGICORE also outperforms a combination of Self-Refine (Madaan et al., 2023) and Self-Consistency by 4.0%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-refinement methods (Madaan et al., 2023) show that models can improve through iterative critique or feedback, often using instructive signals or revision exemplars.”
From GIER: Gap-Driven Self-Refinement for Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“It has been demonstrated that prompting-only setups can work well for stylistic or qualitative improvements (e.g.Madaan et al., 2023;Chen et al., 2023b), but would require external feedback for reasoning tasks (Huang et al., 2023;Shinn et al., 2023;Kim et al., 2023;Madaan et al., 2023).”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“This aligns with recent concerns about the reliability of self-refinement, as Huang et al. (2023) questions the findings of Madaan et al. (2023), suggesting that the claimed improvements may not be robust.”
From Voting or Consensus? Decision-Making in Multi-Agent Debate (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023), as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Meta-Harness is also closely related to methods such as ProTeGi, TextGrad, OPRO, GEPA, AlphaEvolve/OpenEvolve, and Feedback Descent, which iteratively improve prompts or other text artifacts using feedback from prior attempts [38; 31; 53; 51; 1; 35; 43; 26].”
From Meta-Harness: End-to-End Optimization of Model Harnesses (2026) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“Building on early research showing that LLMs can produce useful feedback and iteratively improve outputs (Madaan et al., 2023), recent work has explored using multimodal models as visual critics to ground code generation and refinement.”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-Refine (Madaan et al., 2023) shows diminishing marginal returns across multiple rounds of self-refinement, with each additional iteration providing smaller gains.”
From Can LLMs Self-Correct Table Reasoning Errors? (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-Refine introduced iterative self-critique, demonstrating that models could improve their own outputs through multi-turn prompting loops [Madaan et al., 2023].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Self-correction methods enhance output quality through iterative refinement [6]”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“We observe only marginal improvements on REALINSTRUCT and a performance drop on IFEval using the Self-Refine (Madaan et al., 2023) pipeline, highlighting the limitations of traditional self-correction approaches for the multi-constrained instruction following task.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023)”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“and applying their self-correction prompt on top of model responses from our stronger version of the standard prompting again leads to a decrease in performance.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Table 8 demonstrates that our new prompt, denoted as Standard Prompting (ours) , significantly outperforms the results after self-correction of Madaan et al. (2023)”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“we take the Constrained Generation task in Madaan et al. (2023) as an example, where the task requires models to generate coherent sentences using all 20-30 input concepts.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Madaan et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“Addressing the notable challenges of self-correction in large language models [Madaan et al., 2023, Huang et al., 2023b], AutoMix employs a non-LLM setup for routing and avoids escalating issues such as hallucination and reasoning errors [Dziri et al., 2023].”
From AutoMix: Automatically Mixing Language Models (2023) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“First, 'self-improvement' is not one thing. The term is used for inference-time output revision (Self-Refine [6]), training loops on self-generated data (STaR [7], Self-Rewarding LMs [8]), agents that rewrite their own code (Gödel Agent [9], Darwin Gödel Machine [10]), and systems that autonomously conduct AI research [11]. These differ enormously in ambition and risk profile, yet the literature's”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“A key feature of the Code agent is its self-refinement mechanism [31], which enables iterative improvement of generated code.”
From SafeCodeRL: Security-Constrained Multi-Agent Reinforcement Learning for Trustworthy LLM-Generated IoT/CPS Software (2026) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“many recent studies have investigated the potential of LLMs to correct and refine their outputs, demonstrating that they indeed possess those capabilities (Welleck et al., 2023; Madaan et al., 2023; Shridhar et al., 2023; Ganguli et al., 2023; Wang et al., 2023b; Qi et al., 2023; Yang et al., 2023).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Similar to humans, LLMs do not always generate optimal outputs on their first attempt.”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Reflection-based methods, such as Reflexion [11] and Self-Refine [12], provide critical feedback that allows agents to learn from previous mistakes.”
From Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training (2025) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“Critique-and-refinement is a technique to improve the initial output of an LLM on a specific task.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Madaan et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies demonstrated that LLMs can serve as effective critics, generating detailed and constructive feedback [13, 14].”
From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Prompts used by Madaan et al. (2023) for generating initial responses instruct to generate responses that are not interesting and not very engaging, which are contradicting to the task goal.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, few existing approaches jointly evolve the full set of editable components [16]; most focus on a single component, typically the prompt [32, 50, 20], skills [19, 43], or an in-context playbook [49].”
From Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredIn the citing paper
“Prior studies often simply state their research questions as whether LLMs can self-correct their mis-takes (e.g., Kim et al., 2023;Madaan et al., 2023).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“Few-shot examples for generating initial responses include examples in settings different from the target task (positive to negative), while all few-shot examples for refinement are positive to negative.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“However, most studies in intrinsic self-correction (Madaan et al., 2023;Huang et al., 2024a) use simple prompts for generating feedback, and there is room for improvement.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“As more obvious weak prompts, Self-Refine (Madaan et al., 2023) uses instructions or few-shot examples that do not correctly correspond to the target task only for initial response generation (e.g., providing wrong target labels in few-shot examples), while using appropriate instructions for self-correction, as shown in Table 9 and 10.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
In the citing paper
“For example, the constrained generation task evaluated in Self-Refine (Madaan et al., 2023) is a task to generate a sentence that includes five specified words.We can easily evaluate the correctness by checking whether the five words are included in the generated sentence.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Madaan et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Q: There were nine computers in the server room.Five more computers were installed each day, from monday to thursday.How many computers are now in the server room?
“| This work | SELF-REFINE w/ GPT-3.5 SELF-REFINE w/ ChatGPT | 62.4 75.1 | | | SELF-REFINE w/ GPT-4 | 94.5 |”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Math Reasoning 64.1 64.1 (0) 74.8 75.0 (↑0.2) 92.9 93.1 (↑0.2)”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“| SELF-REFINE wins | 36.0 | 48.0 | 54.0 | | INIT wins | 23.0 | 18.0 | 16.0 | | Both are equal | 41.0 | 50.0 | 30.0 |”
Reported 100 · recomputed 116
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“* Well-known: CSCLE is not a well-known acronym.5/5 * Total score: 23/25”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
Reported 23 · recomputed 24
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“We use greedy decoding with a temperature of 0.7 for all setups.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
“We consider the temperature of both T = 0.0 (greedy) and T = 0.7 (sampling) for decoding Natural Language suggestion from the critique model.We always use a temperature T = 0.0 (greedy) when decoding Programming Language from the code editor.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.3 Results
“Generally, GPT-4+SELF-REFINE performs better than GPT-3.5+SELF-REFINEand ChatGPT+SELF-REFINE across all tasks”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Sentiment Reversal 8.8 30.4 (↑21.6)11.4 43.2 (↑31.8)3.8 36.2 (↑32.4)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“* Well-known: CSCE is not a well-known acronym.2/5 * Total score: 20/25”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Q Acronym Generation
Reported 20 · recomputed 21
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Refine Feedback
“We show that SELF-REFINE outperforms direct generation from strong LLMs like GPT-3.5 (text-davinci-003 and gpt-3.5-turbo;OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“Dialogue Response 36.4 63.6 (↑27.2) 40.1 59.9 (↑19.8)25.4 74.6 (↑49.2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > 3.1 Instantiating SELF-REFINE
“The FEEDBACK-REFINE iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > L.2 Experiments
“Due to budget constraints, we run SELF-REFINE for N = 5 iterations.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
“Table 13: SELF-REFINE results from table 1 with Wilson confidence interval (at 95% confidence interval) and statistical significance.”
§ SELF-REFINE: Iterative Refinement with Self-Feedback > Photosynthesis
“Table 13 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at α= 99% confidence interval) and statistical significance.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2×
“We also experiment with Vicuna-13b (Chiang et al., 2023), a version of LLaMA-13b (Touvron et al., 2023) fine-tuned on conversations sourced from the web.”
BackgroundMethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481 · cited 11×
“Dataset: (Zhang et al., 2015) 1000 review passages x: The food was fantastic.”
BackgroundMethodInfluentialThe Architecture of Complexity
Simon, H. A. (1991) · Springer eBooks · doi:10.1007/978-1-4899-0718-9_31 · Cited by 4,479 · cited 1×
“Iterative self -refinement is a fundamental characteristic of human problem-solving (Simon, 1962; Flower and Hayes, 1981; Amabile, 1983).”
BackgroundChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Interval Estimation for a Binomial Proportion
Brown, L. D., Cai, T., & Dasgupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,746 · cited 1×
“Table 13 shows results from Table 1 with Wilson confidence interval (Brown et al., 2001) (at α= 99% confidence interval) and statistical significance.”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 4×
“5 (text-davinci-003 and gpt-3.5-turbo; OpenAI; Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) by 5-40% absolute improvement.”
BackgroundInfluentialEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Research works printed but not linked: 3
- Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu. 2019. Teaching a black-box learner. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1547-1555. PMLR.
- William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022a. Self-critiquing models for assisting human evaluators.
- Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022a. Peer: A collaborative language model. · DOI · Paper page
Cited by
Sorted by citations
Items 501–600 of 778
Multi-Agent LLM Pipeline for Code Writing: An Experimental Study of Writer-Reviewer Architecture
Ekiz, T. K., & Konyar, M. Z. (2026) · ADBA computer science. · doi:10.69882/adba.cs.2026071
Deconstructing Superintelligence: Identity, Self-Modification and Différance
Perrier, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33195-3_14 · Cited by 1
Yin, X., Chen, Z. Q., Shen, Y., et al. (2026) · Applied Sciences · doi:10.3390/app16157396
InsEt: Making large language models learn from previous rationales
Liao, Z., Liao, J., Tan, Z., & Tang, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133726
Chang, E. Y. (2026) · ACM eBooks · doi:10.1145/3822380.3822386
Wang, S., Fu, Y., & Kim, J. (2026) · Journal of Management in Engineering · doi:10.1061/jmenea.meeng-7498
Zhong, G., Huang, Q., Chang, Z., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7193778
Feedback Granularity in Bounded Repair of LLM-Generated PDDL Domains
Zhang, Y., Li, N., Zhou, Z., & Lu, Y. (2026) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1109/caibda70336.2026.11621110
EvoMod: Evolutionary Optimization of Modular Prompts for Biomedical Relation Extraction
Yang, L., Zhan, J. Z., & Chen, J. (2026) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3807503.3819434
Chang, Y.-T., Chiang, S., Chiu, Y., et al. (2026) · doi:10.2196/preprints.108009
Self-citationAtreya, U., Ansari, M. A., & Saxena, N. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620982
MathTEC: A Multi-Agent Framework for Mathematical Text Error Correction
Li, X. (2026) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs69761.2026.11613421
The intelligence loop and value creation in the age of AI agents
Li, Y., Zeng, M., & Li, A. (2026) · Strategic Entrepreneurship Journal · doi:10.2139/ssrn.6508698
MERIT: A Multi-Signal Repair Framework for Code Translation with Large Language Models
Guo, J. (2026) · International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) · doi:10.1109/icbase70763.2026.11619379
Li, C., & Zhang, M. (2026) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.70360
Ubah, C., Taylor, B., & Kaza, S. (2026) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera69989.2026.11618606
Automated Requirements Assessment and UML Generation from Natural Language
Zhumagaliyev, A., & Kim, H.-J. (2026) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera69989.2026.11618662
Beyond Trajectory: Explicit Case-Based Memory for Decision-Making in Multimodal GUI Agents
Zhang, Y., & Zheng, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_21
TextBFGS: A Case-Based Reasoning Approach to Code Optimization via Error-Operator Retrieval
Zhang, Z., Liao, Y., Chen, C., He, J., Wu, D., Yu, Q., Gao, Y., Yang, J., Zhang, K., Zhong, X., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_19
SigRef: Verification-Driven Reflection for Faithful Paper-to-Code Development
Zhou, M., Yao, Q., Du, L., Wei, L., & Zheng, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31930-2_12
The Human Layer of Agentic AI Memory: What Self-Improving LLM Systems Actually Learn in Production
Parameswaran, A., Hussain, W., & Hossain, M. N. (2026) · Studies in big data · doi:10.1007/978-3-032-31204-4_3
Mozi-TSEIP: Task Self-Evolution in Multi-Agent Systems
Liu, X., Qi, F., Li, X., Chu, D., Yuan, Y., Du, H., & Zhang, R. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-92-0232-4_54
Luo, J., Liu, M., Huang, Z., Ke, Z., & Wang, Q. (2026) · Electronics · doi:10.3390/electronics15153452
Kim, D. K., Lee, Y.-W., & Jeong, J.-S. (2026) · Applied Sciences · doi:10.3390/app16157733
From Prompts to Pressure: Evaluating LLM-driven Agents for GPU Stress-code Generation
GENSALE, A., Esposito, G., Guerrero-Balaguera, J.-D., Condia, J. E. R., Cagliero, L., & Reorda, M. S. (2026) · IEEE International On-Line Testing Symposium (IOLTS) · doi:10.1109/iolts69666.2026.11633814
Vatsal, S., Dubey, H., & Bajaj, A. (2026) · IEEE Access · doi:10.1109/access.2026.3720396
Hong, S., Kong, H., Lee, J., Kim, S., & Yoo, H. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00079
Pioneering multimodal emotion recognition in the era of large models: Towards open vocabularies
Han, J., Gao, Z., Gao, S., et al. (2026) · AI Open · doi:10.1016/j.aiopen.2026.06.001
Uncertainty-Aware Adaptive Debate for Robust and Efficient Large Language Model ReasoningPreprint
Liu, R., Guo, W., Liao, T., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10581777/v1
Zhang, J., Li, Y., Feng, X., Zhang, L., Huang, Z., & Chen, C. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818853
SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition
Tang, J., Yuan, X., Liu, J., Yu, J., Dong, X. M., Chen, L., Teng, Y., Di, S., & Yin, J. P. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817763 · Cited by 14
Li, Y., Dou, Y., Shao, J.-J., Lyu, Y., Tsang, I. W.-H., & Yin, H. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817981
Zhao, Y., Hu, T., & Jiang, S. (2026) · Symmetry · doi:10.3390/sym18081340
Incorporating Reflection Mechanism into Multi-Modal Reasoning for MLLMs
Leng, X. (2026) · International Conference on Electronic Information Technology and Computer Engineering (EITCE) · doi:10.1109/eitce70137.2026.11634616
ARCANA: A Reflective Multi-Agent Program Synthesis Framework for ARC-AGI-2 Reasoning
Zhang, K., Fu, L., Wang, Z., et al. (2026) · International Symposium on Advances in Electrical, Electronics and Computer Engineering (ISAEECE) · doi:10.1109/isaeece70502.2026.11635197
CausalDX: Diagnosing Long-Tail and Cascading Cloud Incidents With LLM-Guided Causal Reasoning
Chang, Z., Wu, Y., Feng, H., Li, Y., Fang, Y., Liu, L., Jin, Z., Huang, D., Yang, X., Chen, P., et al. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3722256
SAGE: A Self-Adaptive Agentic Framework for Execution-Guided Code Generation and Self-RepairPreprint
Lodhiya, H. (2026) · Research Square · doi:10.21203/rs.3.rs-10523195/v1
Hua, L., Zheng, J., Zheng, H., Zhang, L., Zheng, G., & Zhang, L. (2026) · IEEE China Conference on System Simulation Technology and its Applications (CCSSTA) · doi:10.1109/ccssta69471.2026.11635782
Graph-Based Multi-agents for Text-to-SQL
Hùng, P. Q., Thi-Ha, L., Vu, H.-T., Xuan-Hieu, P., & Nguyen, M.-T. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-2600-9_7
Video-Grounded Verification for Long-Horizon Visual Imitation Learning
Chen, Q., Shi, C., Chen, Q., Wu, Y., Zhang, X., Gao, R., Zhang, H., Gao, Z. F., Sun, C., Wu, K., et al. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3723308
An Expert-Guided Architecture for Modular and Multi-Client Multi-Agent Systems
ZHANG, Y., Guillaume, M. P., LELEUX, E., & Krauth, A. (2026) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet65726.2026.11632477
LLM as Automated Theorem Provers for Map-Reduce Code Generation
Simonov, V. S., & Khairetdinov, M. S. (2026) · International Siberian workshops and tutorials on electron devices and materials · doi:10.1109/edm69524.2026.11632200
Agentic AI for Data Quality: Probabilistic Pivot for Data Quality Management
Upadhyay, P., Rajwade, M., Leuva, N., Dhar, S., & Palecek, D. (2026) · International Conference on Advancement in Data Science, E-learning and Information System (ICADEIS) · doi:10.1109/icadeis71120.2026.11644550
Shi, H., Zhang, J., & Zhang, K. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00111 · Cited by 9
Chen, J., Wang, H., & Shao, S. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00156
Can One Agent Restore Another? Multi-Agent Verification Through Independent Constraint SourcesPreprint
Yuan, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10651733/v1
Jeong, C., & Shin, Y. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194026500695 · Cited by 5
Cao, S. (2026) · International Conference on Artificial Intelligence and Electromechanical Automation (AIEA) · doi:10.1109/aiea70743.2026.11632893
Pham, V. T., Phan, M., Quynh, C. B., Phuc, T. C., Duy, N. T., Tien, M. V., & Son, N. N. (2026) · Malaysian Journal of Science and Advanced Technology · doi:10.56532/mjsat.v6i2.684
An LLM-Powered Intelligent Assistant for Learning Management Systems Using RAG and NL2SQL Pipelines
Acharya, S. S., Shenoy, G. S., Shenai, S., GudurNitte, S., & Tushar, D. S. (2026) · International Conference on Distributed Computing and Electrical Circuits and Electronics (ICDCECE) · doi:10.1109/icdcece68961.2026.11646016
Li, B., Guo, Y., Hu, H., Rao, D., & Li, D. (2026) · Energies · doi:10.3390/en19163817
Guan, X., Cao, J., Cao, B., Gao, Q., & Liu, B. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3724403
Gundidza, F., Kikuchi, M., & Ozono, T. (2026) · IEEE Access · doi:10.1109/access.2026.3723798
LLM-Driven Low-Code Task Orchestration for Agile Flexible Manufacturing in Mass Customization
Deng, Y., & Yu, J. (2026) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma69663.2026.11647594
Gueddes, A., Louhichi, B., & Mahjoub, M. A. (2026) · Cluster Computing · doi:10.1007/s10586-026-06436-8
Agentic LLM Compilation of Adaptive Sampling Policies for Low-Power IoT Sensing
Boyouk, N., Behnamian, S., & Dehkordi, M. K. H. (2026) · Sensors · doi:10.3390/s26175519
Eryılmaz, B., Bahn, M., Damm, H., Chu, B., Bender, N., Kleesiek, J., & Nensa, F. (2026) · doi:10.2196/preprints.109219
Tian, Y., Xiao, Q., Wei, H., & Wang, F. (2026) · Electronics · doi:10.3390/electronics15163666
Analysis and Classification of Approaches to the Evolution of LLM-Agents
Равильевна, Ф. Р., & Sychev, A. E. (2026) · International Conference on Neural Networks and Neurotechnologies (NeuroNT) · doi:10.1109/neuront71829.2026.11651389
Akintande, O. J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7295938
DiaVTG: multi-turn reasoning framework for video temporal grounding
Huang, H., Yang, J., Yang, S., & Jin, X. (2026) · The Visual Computer · doi:10.1007/s00371-026-04645-1
Jiménez-Sánchez, A., Rodríguez-Marcos, C., Domínguez-Castro, S., Carrera, A., & Alonso, R. S. (2026) · Electronics · doi:10.3390/electronics15163704
Benchmarking Normative AI Assistants Under Inconsistent Evidence with Paraconsistent Trace Semantics
Ulizko, M., Chernikov, A. V., Tomilov, I., Gusarova, N., & Vatian, A. S. (2026) · AI · doi:10.3390/ai7080321
TRACE-QA: Task-routed constraint elimination for auditable multi-agent question answering
Lu, J., Chen, H., Zhu, Y., Xu, J., & Xu, A. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01141-3 · Cited by 1
Xu, J., Yu, L., Wang, P., et al. (2026) · IEEE Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00065
ReviewTestFuse: Fusing review-guided and direct test generation for deployable regression testing
Zhen, H., & Zhang, Z. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00669-y
Pham, A., White, L. L. A., Watson, K. L., et al. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--61023
Optimization of Large Model Knowledge Extraction Chain-of-Thought for Hydropower Operation Agents
Wan, K., Deng, W., Zhao, Y., & Sun, Y. (2026) · International Conference on IoT, Data Science and Advanced Computing (IDSAC) · doi:10.1109/idsac70973.2026.11650689
SEMAD: Semantic-Enhanced Malicious Package Detection Using LLMs and Prompt EngineeringPreprint
Anh, T. D.-T., Nguyen, H.-L., Truong-Le, T. M., & Vu, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7323943
Toward Causal Reasoning in GeoAI : A Three‐Tier Framework for Spatial Decision Support
Niu, Y., Zhang, Z., Yuan, M., Cheng, T., & Li, W. (2026) · Transactions in GIS · doi:10.1111/tgis.70372 · Cited by 1
Evaluating Feedback by Iterative Repair of Multi-step Solution Documents
Lengfeld, T., Seitz, J., & Timofte, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36033-5_19
A semi-automated LLM-based framework for word sense disambiguation in Serbian
Petalinkar, S., Stanković, R. M., Nešić, M. I., & Krstev, C. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261469292
Recursive Self-Improving LLM Agents with Meta-Cognitive Feedback Loops
Kushwaha, S., Shah, B., Chauhan, Y., Saxena, A. M., & Priya, S. S. (2026) · International Conference on Smart Systems and Inventive Technology (ICSSIT) · doi:10.1109/icssit69151.2026.11656524
Utility-Guided Orchestration for Cost-Efficient Tool-Augmented LLM Services
Liu, B., Zhao, G., Xu, H., Tang, H., Zhu, W., Li, L., & Zhang, Y. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00053
Adaptive Ambiguity Reduction in Crop Disease Diagnosis via an LLM Agent
Ghommat, K., Bourkoukou, O., & Ouaarab, A. (2026) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc71566.2026.11650340
Reinforcement Learning Enhanced LLM Agents for Complex Vehicle Routing Problems
Chen, Y., Yu, Z., Wang, J., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36217-9_21
Ferrer, J., & Chicano, F. (2026) · Expert Systems · doi:10.1111/exsy.70401
Designing Reliable LLM Systems for AI-Assisted Engineering Education
Venkatachalapathy, R. (2026) · IEEE International Conference on Innovation, Ethics & Emerging Tech in Engineering and Computing Education (IE2C) · doi:10.1109/ie2c69620.2026.11663036 · Cited by 3
FTDI: A budget-aware self-healing framework for LLM multi-agent code generation
Men, S., Lu, Z., Luo, H., & Zheng, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134164
Bao, H., Yu, T., Wang, J., Yin, H., Gao, Y., & Mao, Y. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1844470
A Dynamic Evaluation Approach to Repository-Level Code Generation Via LLM
Kuang, F., Liu, D., Wang, Y., Huan, H., Zhao, M., Mi, L., & Zheng, X. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00019
Wu, A., Luo, S.-N., Li, J., & Jin, Y. (2026) · 電腦學刊 · doi:10.63367/199115992026083704006
A Domain-Agnostic Agentic Architecture for Structured Extraction of Engineering Knowledge
Ikechukwu, O. C., Tarkian, M., Nambiar, S., Jonsson, M., & Brax, C. (2026) · Applied Artificial Intelligence · doi:10.1080/08839514.2026.2712892
FlightLLM: few-shot interval forecasting for airline passenger and baggage demand
Gao, Y., Li, C., Zhong, Z., et al. (2026) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2026.105941
Zhang, Y., Zhang, J., Wang, J., & Zhang, S. (2026) · Sensors · doi:10.3390/s26175571
Symbolically Checked Multi-Step Reasoning via Verification-Weighted SearchPreprint
Zhang, Y., Goel, D., Ahmad, H., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7397293
Gao, L., Shen, G., Hu, W., Chen, W., & Chen, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7417887
Self-citationYichen, Y., Dianqing, L., & Meng, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7422696
Stabilizing Graph Explanations Under Spurious Correlations via Self-reflection
Cai, K., Xu, C., Peng, C., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37657-2_29
Kumar, N. A., & Kataria, V. (2026) · Journal of Hospitality and Tourism Insights · doi:10.1108/jhti-12-2025-1458
Sun, F., Tong, Z., Wei, H., Racharak, T., & Nguyen, L.-M. (2026) · IEEE Access · doi:10.1109/access.2026.3731685
AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation
Jin, S., Sun, Z., Xu, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37447-9_6
Phisetpakasit, S., Omondiagbe, O. P., Licorish, S. A., Savarimuthu, T. B. R., & Nazar, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7435043
Jia, L., Zhou, J., He, Y., & Sun, G. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3994-8_27
So, C.-C., Lee, C., Yang, D., Loh, A. W. K., & Wu, C.-F. (2026) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2026.3731978
Durgun, S., & Genç, O. (2026) · Osmaniye Korkut Ata Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.47495/okufbed.1776290
Debugging Requirements Interview Scripts: A Framework for Quality Assessment
Tefur, A. E.-M., Fotrousi, F., Ferrari, A., Spoletini, P., & Maalej, W. (2026) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re68928.2026.00007
Rebuilding AI from the Small: The Path Toward Democratized Intelligence
Porta, A. D. (2026) · IEEE/ACM International Conference on Software Engineering (ICSE) · doi:10.1145/3774748.3787626
Metadata
- Authors
- Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 46534-46594
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.17651
- OpenAlex
- W7133236450
- Cited by
- 2,314
- References
- 52
- Citations by year
- 2026: 1329 · 2025: 460 · 2024: 338 · 2023: 186
- FWCI
- 260.57
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- iterative refinement, self-feedback, large language models, mathematical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
