Self-critiquing models for assisting human evaluators
Open accessAbstract
We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own self-critiques as feedback, refining their own summaries into better ones. Finally, we motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability. Our measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results are a proof of concept for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. We release our training datasets, as well as samples from our critique assistance experiments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors' format of debate is similar to a fixed point of self-critique and generalizes self-critique.
“Our format of debate is similar to a fixed point of self-critique, where a model critiques its own answers, then critiques those critiques , etc.; in this way, free-form textual debate generalizes self-critique”
From Debate Helps Supervise Unreliable Experts (2023) · cites this paper as Saunders et al., 2022
Section 4.2 (pp. 9–10) describes models critiquing their own answers, supporting the self-critique part of the comparison. The cited paper does not establish an iterative process of critiquing those critiques. The fixed-point analogy and claim that debate generalizes self-critique are the citing authors’ own framing, not findings established by this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredOnly GPT-4 starts to afford more reliable self-evaluation and self-refinement.
“while only GPT-4 (OpenAI, 2023a) starts to afford more reliable self-evaluation (Saunders et al., 2022; Shinn et al., 2023; Yao et al., 2023) and self-refinement (Madaan et al., 2023; Chen et al., 2023b).”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Saunders et al., 2022
Section 4.2 (p. 10) reports that more capable models are significantly better at self-critiquing, supporting the claim that capability can improve self-evaluation. It does not establish that only GPT-4 affords more reliable self-evaluation; Section 7.1 (p. 18) says InstructGPT models can already critique well. The Saunders citation is attached to self-evaluation, not self-refinement. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe self-feedback generated internally by LLMs is limited and task-specific relative to feedback from humans.
“LLM's intrinsic self-feedback has limited and task-specific performance compared to human feedback (Saunders et al., 2022)”
From CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023) · cites this paper as Saunders et al., 2022
Section 4 (p. 8) says the model’s critiques were significantly less helpful than humans’ critiques, supporting a limited comparison on the studied critique task. Section 7.2 (p. 19) cautions that positive results on an easier task might not reproduce on harder tasks. The paper does not establish a general comparison between LLM self-feedback and human feedback across tasks. (LLM, conf=0.82)
Experimental — accuracy not yet measuredChain-of-Thought prompting improves interpretability, but it still follows a single reasoning path in which errors can propagate without independent verification.
“Chain-of-Thought prompting [25] improves interpretability, but it still follows a single reasoning path in which errors can propagate without independent verification [26], [27].”
From Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats (2026) · cites this paper as [27]
The Motivation section (p. 2) says unchecked model outputs can appear sound while containing flaws, and the Theory background (p. 28) says cross-checking a second prover can prevent deceptive strategies. These passages support the concern about absent independent checking, but they do not discuss Chain-of-Thought prompting, a single reasoning path, or errors propagating along one. (LLM, conf=0.80)
Experimental — accuracy not yet measuredSome approaches are validated only on small, specific constraint sets, limiting their general applicability.
“Some approaches are validated only on small, specific constraint sets, limiting their general applicability (Schick et al., 2023; Saunders et al., 2022; Lee et al., 2024; Mu et al., 2023; Yang et al., 2022; Gou et al., 2024; Varshney et al., 2023).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Saunders et al., 2022
The Limitations section (p. 19) says the study’s base task was relatively easy to evaluate and that positive results might not reproduce on harder tasks. This supports a concern about general applicability, but the paper does not characterize its validation as being limited to small, specific constraint sets. (LLM, conf=0.82)
Experimental — accuracy not yet measured
The closest predecessor to CriticGPT is Saunders et al. [26], which also directly trains models to produce critiques.
“The closest predecessor to CriticGPT is Saunders et al. [26] which also directly trains models to produce critiques.”
From LLM Critics Help Catch LLM Bugs (2024) · cites this paper as [26]
Unstable generation-discrimination-critique gaps are especially prominent in tasks requiring external knowledge or complex reasoning, such as QA, commonsense reasoning, and math reasoning.
“We find such unstable generationdiscrimination-critique gaps (Saunders et al., 2022) becomes particularly prominent in tasks that necessitate external knowledge or intricate reasoning, such as QA, Commonsense Reasoning, and Math reasoning.”
From CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023) · cites this paper as Saunders et al., 2022
Saunders et al. (2022)'s AI critiques inspired untrusted justifications.
“Saunders et al. (2022)'s AI critiques inspired untrusted justifications.”
From AI Control: Improving Safety Despite Intentional Subversion (2023) · cites this paper as Saunders et al. (2022)
Critique-and-refinement is a technique to improve the initial output of an LLM on a specific task.
“Critique-and-refinement is a technique to improve the initial output of an LLM on a specific task.”
From Debating with More Persuasive LLMs Leads to More Truthful Answers (2024) · cites this paper as Saunders et al., 2022
Saunders et al. (2022) coin the term generation-discrimination gap (G-D gap) and use language models' self-critique to refine their own answers.
“Saunders et al. (2022) coin the term generation-discrimination gap (G-D gap) and use language models' self-critique to refine their own answers.”
From Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023) · cites this paper as Saunders et al., 2022
Evidence from several directions suggests that LLMs sometimes 'know' more than they 'say'.
“Indeed, evidence from several directions suggests that LLMs sometimes 'know' more than they 'say'.”
From Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023) · cites this paper as Saunders et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 50
- Authors
- Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., & Leike, J.
- ORCID
- Wu, Jeff
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2206.05802
- OpenAlex
- W4282968607
- Cited by
- 50
- References
- 0
- Citations by year
- 2026: 1 · 2025: 15 · 2024: 16 · 2023: 18
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- self-criticism, large language models, behavior cloning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
