CoChat
Sign Up
PreprintCited by 29Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Understanding the Effects of RLHF on LLM Generalisation and Diversity

Open access
Kirk, Robert, Mediratta, Ishita, Nalmpantis, Christoforos, Luketina, Jelena, Hambro, Eric, Grefenstette, Edward, Raileanu, Roberta·2023·arXiv
DOI 10.48550/arxiv.2310.06452arXiv 2310.06452
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper analyzes how supervised fine-tuning (SFT), reward modeling, and RLHF affect out-of-distribution (OOD) generalisation and output diversity in LLMs. It finds RLHF generalises better than SFT, especially under larger distribution shifts, but substantially reduces output diversity, revealing a generalisation–diversity tradeoff.

Key findings

  • RLHF improves in-distribution and OOD performance over SFT; on the harder Sequential Instructions OOD task, RLHF generalises much better, and GPT-4 head-to-head winrates for RLHF and Bo16 vs SFT improve by approximately 3.5% from ID to AlpacaEval OOD.
  • In summarisation, Bo16 outperforms RLHF, which outperforms SFT both ID and OOD; the reward model generalised near-perfectly to CNN/DailyMail preference data, with ID and OOD accuracy of 75.8 and 71.6 respectively.
  • RLHF significantly reduces per-input output diversity compared with SFT across EAD, Sentence-BERT, and NLI metrics; across-input diversity differences are smaller, but still show evidence of mode collapse. For example, in OPT summarisation per-input EAD at 125M was 0.15 for RLHF vs 0.81 for SFT.
  • Increasing the KL penalty did not trade off generalisation for diversity: higher KL penalties led to worse performance and lower per-input diversity.

Methods

  • Models: LLaMa 7B for main experiments, plus OPT models across five sizes in appendix experiments; tasks were Reddit post summarisation (TL;DR, with CNN/DailyMail as OOD) and instruction following (AlpacaFarm Self-Instruct, AlpacaEval, and a generated Sequential Instructions OOD set).
  • Training: SFT, reward modeling, RLHF with PPO and a KL penalty (β_KL = 0.05), and Best-of-N sampling with N = 16 and temperature 0.7.
  • Evaluation: GPT-4 as a simulated human evaluator for preference vs reference and head-to-head winrates; diversity measured with expectation-adjusted distinct N-grams (EAD), Sentence-BERT cosine similarity, and NLI diversity, using K = 16 outputs for N = 500 inputs.

Limitations

  • The paper states it provides no theoretical explanation for the generalisation and diversity effects, evaluates only SFT, RLHF, and BoN among possible fine-tuning methods, and performs no human evaluation, relying on automatic metrics validated as proxies.
  • The text also makes evident that diversity results for instruction following were not meaningful, likely because the diversity metrics are designed for shorter outputs and RLHF outputs tend to be longer, confounding length-sensitive metrics.
  • Model selection used in-distribution validation sets, not OOD data, which the authors note matches a real-world setting but may affect reported OOD comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.06452

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 24 cited sources checked · 4 supported · 2 flagged · accuracy not yet measured

  • Premisecitation

    Freezing 80% of layers can still achieve good results.

    “as more recent work (Glaese et al., 2022;Menick et al., 2022) has shown that good results can still be achieved, and training is much more computationally efficient.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Glaese et al., 2022 source
    • cites Menick et al., 2022 source

      The Training procedure (F.1, p. 35) says this paper froze 60% of its model's weights to reduce memory usage, while a footnote attributes freezing 80% of layers to Perez et al. (2022). The Results (3.3, p. 13) report strong answer quality, but our copy does not establish that freezing 80% achieved those results or made training much more computationally efficient. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    Ramamurthy et al. (2022) only investigated relatively small models (220 million parameters).

    “their work only investigates relatively small models (220 million parameters)”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ramamurthy et al. (2022) source
  • Premisecitation

    The dataset from Stiennon et al. (2022) (filtered from Nallapati et al., 2016) comes with train, validation and test splits.

    “The dataset we use from (Stiennon et al., 2022) (filtered from (Nallapati et al., 2016)) comes with train, validation and test splits, which we use throughout our work.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Stiennon et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Nallapati et al., 2016 source
  • Premisecitation

    The diversity metrics (distinct N-grams, Sentence-BERT, NLI) align well with human diversity evaluations and underlying diversity generators.

    “All of these metrics have been shown to align well with human diversity evaluations and with underlying diversity generators by Tevet & Berant (2021).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Tevet & Berant (2021) source

      The paper’s Set B overview reports that n-gram metrics correlate well with decoding parameters, and its Results section finds that neural metrics—especially Sentence-BERT—perform better on content diversity. But the Results also say n-gram metrics perform poorly on content diversity and humans substantially outperform every automatic metric; the claim that all these metrics align well with human evaluations is therefore overstated. The paper identifies an NLI-trained Sentence-BERT model, not a separately evaluated NLI metric. Page numbers were not available in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data

    For SFT models, there is a slight downward trend in the average generalisation gap across dataset splits as model size increases.

    “There is a slight downward trend in the average generalisation gap across dataset splits as model size increases, implying that larger models fine-tuned with SFT generalise better (in terms of automated metrics).”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 16
  • May combine two claims. 29

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ramamurthy et al. (2022) did not use reward functions trained with human feedback, instead using mostly hard-coded reward functions from the literature.

    “does not use reward functions trained with human feedback, instead using mostly hard-coded reward functions from the literature.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ramamurthy et al. (2022) source

      Section 5.2 (p. 8) directly contradicts the claim that the authors did not use rewards trained with human feedback: they collected human preferences, trained a reward model on them, and used it as a reward function. Section 3.2 (p. 4) also lists literature-derived metrics, but their use does not support the categorical claim that human-feedback-trained rewards were absent. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Ramamurthy et al. (2022) did not evaluate out-of-distribution performance or diversity.

    “does not evaluate OOD performance or diversity”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ramamurthy et al. (2022) source

      The Results and Discussion (p. 22) explicitly says the IMDB evaluations report diversity metrics, and the results table (p. 55) gives diversity scores. This contradicts the claim that the paper did not evaluate diversity. The passages reviewed do not establish whether it evaluated out-of-distribution performance. (LLM, conf=0.92)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Training a single model with policy and value head is much more computationally efficient than training separate policy and value functions.

    “For RLHF, we train a single model with policy and value head, rather than separate policy and value functions.This is much more computationally efficient, and follows other recent work that still achieves impressive results (Glaese et al., 2022).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Glaese et al., 2022 source

      Appendix E (p. 45) says the authors share parameters between the policy and value functions to reduce memory usage, and describes a separate value-function MLP using the shared transformer representation. Section 2.8 (p. 13) likewise describes a shared trunk with separate heads. The copy does not establish that this is much more computationally efficient than training separate policy and value functions; it identifies a memory-saving benefit instead. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Ramamurthy et al. (2022) demonstrate that reinforcement learning generally outperforms supervised fine-tuning.

    “Ramamurthy et al. (2022) demonstrate that RL generally outperforms SFT”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ramamurthy et al. (2022) source

      Section 5.1 (p. 6) reports that combining RL with supervised training usually performs best, including compared with supervised training alone. But its human-evaluation ranking places supervised training above RL-only methods. The claim omits the crucial supervised warm-start qualification and overstates the result as a general advantage for RL over SFT. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    GPT-4 accurately simulates human preferences in both summarisation and instruction following.

    “Previous work has shown that GPT-4 accurately simulates human preferences in both summarisation (Rafailov et al., 2023) and instruction following (Dubois et al., 2023;Zheng et al., 2023), so we use it as the main performance metric in both tasks.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Rafailov et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Dubois et al., 2023 source

      The Basic GPT-4 prompt design section reports that GPT-4 agrees with human annotators at a rate similar to agreement between humans (65% versus 66%) on the paper’s instruction-following preferences. But it also says a single GPT-4 prompt fails to capture variability in human annotations and can produce qualitatively different results, so the unqualified claim that it accurately simulates human preferences overstates the finding. The summarisation portion is attributed to a different citation, not Dubois et al. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Zheng et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Ramamurthy et al. (2022) demonstrate that the combination of reinforcement learning and supervised fine-tuning performs the best.

    “but that their combination performs the best”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ramamurthy et al. (2022) source

      Section 5.1 (p. 6) says that using reinforcement learning and supervised learning together generally works best, with combined models usually outperforming either approach alone. The claim omits the paper’s stated exception: on DailyDialog, RL-only models outperformed the supervised-plus-RL models. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    In the OPT generalisation experiments, BoN generalises better than RLHF, which generalises better than SFT, according to proxy RM score averaged over dataset sizes.

    “The important result here is that BoN generalises better than RLHF, which generalises better than SFT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    BoN scales better than RLHF, eventually outperforming it for models with more than 2.7b parameters.

    “We see that at middling model sizes, RLHF outperforms BoN, but BoN scales better than RLHF, eventually outperforming it for models with more than 2.7b parameters.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both RLHF and Bo16 winrates improve vs SFT by approximately 3.5% from ID to AlpacaEval OOD.

    “These results show that both RLHF and Bo16 winrates improves vs SFT by approximately 3.5% from ID to AlpacaEval OOD.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    PPO training has converged by approximately 250 PPO training steps.

    “PPO training has converged by approximately 250 PPO training steps, and so we terminated training early to save compute.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RLHF has lower output diversity than SFT, both per-input and across-input.

    “We find that RLHF has stronger generalisation performance but lower output diversity than SFT, demonstrating a tension between these two desirable properties in current LLM fine-tuning techniques.”
    • Also stated in: § Acknowledgements · plain
      “RLHF produces more performant models both in-distribution and out-of-distribution, but at the cost of lower output diversity, both per-input and across-input.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There is a slight downward trend for diversity in SFT models as model size increases.

    “We also see that diversity does not seem to change much with model size, apart from a slight downward trend for diversity in SFT models as model size increases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The relationships and sentiment splits produce negligible generalisation gap for the proxy RM score while the length split is more difficult.

    “The relationships and sentiment splits produce negligible generalisation gap for the proxy RM score while the length split is more difficult.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    At smaller model sizes increasing the number of parameters leads to improvements in SFT performance but not RM performance, while at larger model sizes increasing the number of parameters leads to improvements in both SFT and RM performance, but both to a lesser extent.

    “At smaller model sizes increasing the number of parameters leads to improvements in SFT performance but not RM performance, while at larger model sizes increasing the number of parameters leads to improvements in both SFT and RM performance, but both to a lesser extent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In summarisation, BoN outperforms RLHF on both ID and OOD GPT-4 PvR.

    “Bo16 outperforms RLHF which outperforms SFT, both ID and OOD.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    For SFT and RLHF training, there was not much variance with seeds, but for RM training there was more variance.

    “In general for SFT and RLHF we did not see much variance with seeds, but we did in RM training, matching prior work (Stiennon et al., 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Stiennon et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    For the NLI score, as RLHF model size increases the score increases, eventually reaching the diversity of SFT models.

    “For the NLI score, which is a proxy for logical diversity, we see that as RLHF model size increases the score increases, eventually reaching the diversity of SFT models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The NLI score is a proxy for logical diversity.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    For models with more than 125 million parameters, both SFT and RLHF produce policies with very similar scores.

    “for models with more than 125 million parameters both SFT and RLHF produce policies with very similar scores.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The reward model's ID and OOD accuracy on the CNNDM preference dataset is 75.8 and 71.6 respectively.

    “ID and OOD accuracy is 75.8 and 71.6 respectively”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In OPT models, BoN outperforms RLHF which outperforms SFT at the largest model size, and the ordering holds OOD for 3 different splits of the training dataset.

    “These results show similar trends: BoN outperforms RLHF which in turn outperforms SFT at the largest model size, and the ordering holds OOD for 3 different splits of the training dataset.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Freezing the first 80% of layers and the embedding and unembedding layers causes a drop in performance for smaller OPT models, but the drop is not catastrophic (equivalent to about a drop in model size among the three model sizes used).

    “In Table 14 we show the drop in performance for smaller OPT models between freezing 80% of the layers and not freezing.There is a drop of performance, but it is not catastrophic (equivalent to about a drop in model size among the three model sizes used), which justifies the tradeoff of training models with partially frozen weights.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The GPT-4 evaluator has good agreement with expert human labellers both at an aggregate level and at a per-example level.

    “Table 1 shows the preference rating for GPT-4 and human labellers for each dataset and model, and the agreement between labellers and GPT-4, and we see that both at an aggregate level and at a per-example level our GPT-4 evaluator has good agreement with expert labellers.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human labellers used in the second validation are expert labellers.

  • May combine two claims.

    Findingown data

    In instruction following, RLHF and Bo16 both outperform SFT.

    “Similar to summarisation, we see that RLHF and Bo16 both outperform SFT, but here RLHF outperforms Bo16 across all datasets, in contrast to the summarisation task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    SFT comes out worst in the proxy RM comparison, scaling worse than BoN and the same as RLHF, and having worse absolute performance and generalisation.

    “SFT comes out worst in this comparison, both scaling worse than BoN and the same as RLHF, and having worse absolute performance and generalisation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This experiment highlights the importance of training a reward model on human feedback and using it to select the best outputs at test time, potentially after fine-tuning the model.

    “This experiment highlights the importance of training a reward model on human feedback and using it to select the best outputs at test time, potentially after fine-tuning the model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The drop in across-input diversity cannot be explained purely by the use of the reward model, as BoN has similar or higher across-input diversity than SFT for the first two metrics.

    “The drop in across-input diversity cannot be explained purely by the use of the reward model, as BoN has similar or higher across-input diversity that SFT for the first two metrics.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Evaluating BoN as well as SFT and RLHF enables uncovering whether differences between RLHF and SFT are due to the use of a reward model or the type of optimisation applied.

    “Evaluating BoN as well as SFT and RLHF enables us to uncover whether differences between RLHF and SFT are due to the use of a reward model or the type of optimisation applied.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Formulating the reward model as described allows it to (in theory) learn to predict any transitive ranking over possible outputs while maintaining a type signature suitable for use as a reward function in an RL context.

    “Formulating the RM in this way means it can (in theory) learn to predict any transitive ranking over possible outputs, while still maintaining a type signature suitable for use as a reward function in an RL context”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductreasoning

    The reward model trained with a different random seed and data distribution serves as a held-out automated evaluation of models and a good proxy for human preferences.

    “As this reward model is trained with a different random seed and a different data distribution, it serves as a held-out automated evaluation of models, and a good proxy for human preferences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study evaluates models only on automatic metrics and does not perform human evaluation.

    “Finally, we only evaluate our models on automatic metrics and do not perform any human evaluation (although we validate that our metrics align well with human preferences in Appendix D.1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Low NLI score for smaller models implies they have a tendency to make logically consistent claims in their outputs on top of producing outputs about certain topics of content.

    “Low NLI score for smaller models implies they have a tendency to make logically consistent claims in their outputs on top of producing outputs about certain topics of content.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The NLI score is a proxy for logical diversity.

  • May combine two claims.

    Interpretationown data

    The results are robust across different evaluation metrics and base models.

    “This shows our results are robust across different evaluation metrics and base models.”
    • Also stated in: § 6 EXPERIMENTAL RESULTS · plain
      “These results show similar trends: BoN outperforms RLHF which in turn outperforms SFT at the largest model size, and the ordering holds OOD for 3 different splits of the training dataset.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Using a range of distribution-shift splits gives a more robust measure of how well policies trained with different methods generalise under distribution shifts.

    “While we do not expect these splits to capture the full range of distribution shifts models may experience when deployed, using a range of splits will give us a more robust measure of how well the policies trained with different methods generalise under distribution shifts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Models fine-tuned with RLHF have been observed to be much better in practice when interacting with users.

    “which potentially explains why models fine-tuned with RLHF have been observed to be much better in practice when interacting with users (Touvron et al., 2023b;Ouyang et al., 2022, inter alia)”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Touvron et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ouyang et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The RLHF pipeline (SFT followed by RLHF) is the standard approach used in the literature and in deployed products that use RLHF.

    “We call the process of doing SFT followed by RLHF”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites OpenAI, 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Anthropic, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    An off-the-shelf sentiment classifier (Loria, 2013) can be used to measure the sentiment of posts.

    “for the sentiment split, we use an off-the-shelf sentiment classifier (Loria, 2013), and the training set consists of summaries with sentiment less than the median sentiment in the dataset”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Loria, 2013

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    For combinations where the learning rate was varied, there was not much variation in performance on the metrics measured, so the choices of single learning rates for some combinations are not expected to affect the results.

    “For the combinations where we did vary the learning rate we did not see much variation in performance on the metrics we measured, so we do not expect these choices to affect the results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    BoN will continue to be a useful yet simple technique.

    “We see that BoN sampling does not see diminishing returns as model size increases, implying it will continue to be a useful yet simple technique.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The GPT-4 evaluator has good agreement with expert labellers both at an aggregate level and at a per-example level.

    “We see that both at an aggregate level and at a per-example level our GPT-4 evaluator has good agreement with expert labellers.”
    Quote in the paperNot a substantial claim
39 of 39

  • The paper relies on this without support.

    There is little understanding about how each component of the RLHF pipeline contributes to the behaviour of the final model.

    “While this pipeline has been used to seemingly great success, there is little understanding about how each component contributes to the behaviour of the final model.”

  • The paper relies on this without support.

    No rigorous analysis has been done of the effects of different parts of the RLHF pipeline on output diversity across different tasks.

    “However, no rigorous analysis has been done of the effects of different parts of the RLHF pipeline on output diversity across different tasks.”

  • The paper relies on this without support.

    Prior work on diversity has been limited to evaluating diversity using simple token-level metrics such as BLEU and on use cases which are not as common in practice.

    “In addition, in contrast with our paper, prior work has been limited to evaluating diversity using simple token-level metrics such as BLEU (Papineni et al., 2002) and on use cases which are not as common in practice.”

  • The paper relies on this without support.

    The maximum inter-annotator agreement in the CNNDM preference dataset is lower than in TL;DR.

    “considering that the maximum inter-annotator agreement in the CNNDM preference dataset is lower than TL;DR (and hence the maximum accuracy attainable is lower)”

  • The paper relies on this without support.

    RLHF is most widely applied in settings where we do not have a hard-coded reward function, and hence need to learn one from human data.

    “RLHF is most widely applied in settings where we do not have a hard-coded reward function, and hence need to learn one from human data.”

  • The paper relies on this without support.

    Other fine-tuning methods (Chain of Hindsight, CoOp CARP, RRHF, HIR, ILF) are not used by most large-scale systems being deployed currently.

    “they are not used by most large-scale systems being deployed currently”

  • The paper relies on this without support.

    The human labellers used in the second validation are expert labellers.

    “our GPT-4 evaluator has good agreement with expert labellers”

  • The paper relies on this without support.

    The Sent BERT score is a proxy for diverse content and semantics.

    “However, for the Sent BERT score, which is a proxy for diverse content and semantics, RLHF models are consistently less diverse than SFT models.”

  • The paper relies on this without support.

    The NLI score is a proxy for logical diversity.

    “For the NLI score, which is a proxy for logical diversity, we see that as RLHF model size increases the score increases, eventually reaching the diversity of SFT models.”

  • The paper relies on this without support.

    The three policy types investigated (SFT, BoN, and RLHF) are representative of existing language model fine-tuning techniques.

    “This reinforces the inherent tradeoff between generalisation and diversity present in existing language model fine-tuning techniques.”

  • The paper relies on this without support.

    A performance drop that is not catastrophic is acceptable and justifies using model freezing.

    “The results are shown in Table 14, and show that while performance drops for the models evaluated in the experiment, the drop is not catastrophic, justifying the use of model freezing.”

  • The paper relies on this without support.

    GPT-4 can serve as a valid simulated human evaluator for assessing out-of-distribution generalisation.

    “In this work, we fine-tune large language models (LLMs) with three different techniques (SFT, BoN, and RLHF), and evaluate their out-of-distribution generalisation (using GPT-4 as a simulated human evaluator) and output diversity (using a range of metrics from the literature).”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GPT-4's conservative behavior is interpreted as a shortcut learned during RLHF to reduce output diversity and mitigate hallucinations.

    “We interpret this as a shortcut behavior learned by GPT-4 during aligning with human preferences stage training (e.g., RLHF (Ouyang et al., 2022)) to reduce output diversity to mitigate hallucinations (Kirk et al., 2023).”

    From MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025) · cites this paper as Kirk et al., 2023

    Section 6 (Experimental Results; page unavailable) reports that RLHF reduces output diversity in the models studied. But Section 5.1 uses GPT-4 as an evaluator, not as a model whose conservative behavior is examined. The cited paper does not attribute reduced diversity to a shortcut learned by GPT-4 or say that it mitigates hallucinations. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Low entropy in RLHF models challenges generating diverse responses.

    “Another characteristic of RLHF models is their low entropy in output distribution [12], which challenges generating diverse responses [80].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [80]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure 13 caption describes different policies from the KL sweep

    Minor

    The text introduces Figures 12 and 13 as results for different KL penalty coefficients on RLHF models, whereas Figure 13's caption says it compares RLHF, Bo16 and SFT policies.

  • Figure 11 identifies the wrong source of summarisation references

    Minor

    The Figure 11 caption identifies its summarisation reference outputs as coming from text-davinci-003, but the evaluation-methods section says summarisation references are produced by humans and text-davinci-003 references are used for instruction following.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Tuomas Haarnoja. Acquiring Diverse Robot Skills via Maximum Entropy Deep Reinforcement Learning. PhD thesis, UC Berkeley, 2018. URL https://escholarship.org/uc/item/ 25g6573w.

Cited by

Sorted by citations

Related

Authors
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R.
Affiliations
  • University College London (GB)
  • University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.06452
arXiv
2310.06452
OpenAlex
W4387561082
Semantic Scholar
cb3968152f7d93f53d24b00279a90d5071ddc85a
Cited by
17
References
66
Citations by year
2026: 1 · 2025: 14 · 2024: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
reinforcement learning from human feedback, supervised fine-tuning, out-of-distribution generalization, output diversity, distribution shift, summarization, instruction following, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review