RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

“While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”

“Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when us-ing PaLM 2 XS - a trend consistent with scaling behaviors observed in other work (Kaplan et al., 2020).”

“Despite being the focus of considerable research effort (Ouyang et al., 2022; Bai et al., 2022b; Lee et al., 2023; Tunstall et al., 2023; Khalifa et al., 2021), there is a lack of consensus regarding the optimal approach to achieve this goal.”

“Several studies have also incorporated swapping operations in self-alignment (Lee et al., 2023; Sun et al., 2024; Lee et al., 2024) to obtain more accurate pairwise feed-back from the judge LLM. Zhu et al. (2024a) proposed a CoT-like prompting technique to mitigate the positional bias by asking the…”

“Extensions such as language model feedback (RLAIF) could be a viable alternative to human feedback (Bai et al., 2022b; Lee et al., 2023; Pang et al., 2023).”

“This significant reduction in token consumption translates to a substantial decrease in training and inference resource requirements.”

“Reinforcement learning from AI feedback on the other hand, directly connects a pretrained and well-aligned model to the LLM and helps it to learn from larger and more aligned models [138].”

“Some recent research utilizes Reinforcement Learning from AI Feedback (RLAIF) [32, 62] as an alternative to RLHF for model fine-tuning.”

“(3) Value alignment on human-annotated or AI-annotated preference dataset where humans’ value is injected into the LLMs [1, 39, 56].”

“LLM evaluation metrics are criteria used to quantify the performance of LLM systems in aspects such as correctness of the answers, semantic similarity, and hallucination [50].”

“Techniques like Reinforcement Learning from AI Feedback (RLAIF) (Lee et al. 2023) utilize AI feedback to allow agents to self-improve without expensive human labels.”

“2) Soft Prompt: Soft prompt methods involve appending a sequence of trainable continuous vectors, known as soft prompts, to the input of pre-trained language models.”

“Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022b; Lee et al., 2023), is the leading approach to adapt LLMs towards these complex, often implicitly-defined goals.”

“Despite recent advancements in reinforcement learning (RL) (Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2023; Sun et al., 2023b) and DPO (Rafailov et al., 2024; Chen et al., 2024b; Hosseini et al., 2024), which have been effective in guiding LLMs towards generating more honest, helpful, and…”

“Incorporating human feedback through the training of reward models and applying reinforcement learning in the fine-tuning step [Lee et al. 2023b; Ouyang et al. 2022] has significantly enhanced the generation of human-like language that is aligned with social norms and goals and has led to the…”

“To address these issues, methods that use AI feedback [9–12] have recently gained popularity, most prominently Constitutional AI [10].”

“This”

“For task examples see Figure 5 and Figure 6 for Reddit TL;DR and AnthropicHH respectively.”

“We hope that our survey provides a solid general overview for practitioners as a road map for future works.”

“However, obtaining high-quality human feedback data is expensive and several works such as [35, 2, 25] study the usage of AI feedback to reduce reliance on human input.”

“For simplicity, the instruction x is omitted in the state notation ( x, a 1: t ) thereafter when no ambiguity arises.”

“Wang et. al [21] and Lee et. al [37] observed the position bias when using GPT-4 [11] and PaLM 2 [38] as the judge for the pairwise comparison between candidate LLMs.”

“RLHF offers the most straight-forward approach to achieving this goal.”

“…BoN-RLB, and BoN-RLB(P) against several baselines: (1) STaR from Remark C.3, which uses self-training over correctly generated responses; (2) RL (Lee et al., 2023) with verifier feedback (RL-V); (3) RL with environment feedback (RL-S); (4) standard SFT of the base policy (SFT, N ′ = 1 ); and…”

“…preferences (e.g., selecting As the policy is trained, the generated trajectories become increasingly out-of-distribution from the dataset the RM was trained on, leading to suboptimal performance (Bai et al., 2022a). selecting the better output) or assigns scores based on task-specific…”

“5 to rate the alignment of the LLMs’ responses with respect to the instruction’s coding preference, resulting in annotations comprising a numerical rating and a textual rationale for the rating.”

“Like recent literature (Zheng et al., 2023a; Guo et al., 2024; Lee et al., 2023), we employ LLMs as a judge for these two qualities.”

“Note that safety issues usually caused by the distribution of unseen data (adversarial input) (Wei et al., 2023; Li et al., 2024b), which is elaborated in §6.4.”

“However, expanding the range of knowledge transferred from the teacher model, including feedback on the student’s outputs (Lee et al., 2023) and feature knowledge (Gu et al., 2024), can offer additional benefits.”

“These approaches typically start with an initial policy model obtained by conducting SFT on a target task ( π SFT ), before performing RL tuning using contrastive preference examples (most commonly collected through human feedback [8] or a similar proxy like LLM-generated feedback [37]).”

“With the promising capability of the instruction following via reinforcement learning from human feedback (RLHF) [2, 5, 11], developers extend LLMs, e.g., ChatGPT with a Function Call feature, to the tool-use domain.”

“Reinforcement Learning from AI Feedback (RLAIF), using AI labelers as proxies, offers an alternative (Yu et al., 2024; Lee et al., 2023).”

“On the other hand, explicit reward models can easily be regularized and understood—regardless of whether they are Bradley-Terry models [4], margin-based ranking models [40], or simply any other kind of function that correlates well with human preferences [31; 17].”

“RLHF aligns the linguistic space of LLMs with human values, using a set of human-ranked candidates of given prompts [7, 19, 41, 43, 55, 70, 91].”

“Both types of input-based methods can be used not only to produce desired outputs or to transfer styles; they can also be used to better analyze what it is inside the network [170, 177].”

“Recent advances in RLHF (Ouyang et al., 2022), RLAIF (Lee et al., 2024b) and preference learning (Rafailov et al., 2023) research have identified the benefits of on-policy data for model alignment (Xu et al., 2024b; Tajwar et al., 2024; Zhang et al., 2024c; Guo et al., 2024b; Liu et al., 2024d;…”

“Finetuning LLMs. Methods for finetuning LLMs, including Code-LLMs are categorized into supervised finetuning (SFT), instruction finetuning [64], and RL-based finetuning [28, 44, 49].”

“This self-critique mechanism enables the model to learn from AI-generated feedback specifically tailored to mathematical content (Bai et al., 2022; Lee et al., 2023).”

“…learning has become an essential technology for enhancing the capabilities of MLLMs. Early research primarily focused on Reinforcement Learning from Human Feedback(RLHF) [26, 27, 28, 29], which aligns the outputs of multimodal models with human preferences by incorporating human feedback signals.”

“They first describe the evaluation criteria and value orientation of human preferences to the model and then ask it to assess whether the response matches these standards (Bai et al., 2022b; Lee et al., 2023).”

“Models such as DNABERT [56] have advanced gene annotation, while AlphaFold [5] has achieved groundbreaking success in protein structure prediction.”

“…(SFT), reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022), and other derivatives (Rafailov et al., 2024; Bai et al., 2022b; Lee et al., 2023; Gulcehre et al., 2023; Dong et al., 2023; Xiong et al., 2024; Li et al., 2023; Zhou et al., 2023, 2025; Ji et al., 2024a), are the…”

“Training Data Strategy Step-wise Reflection demonstrated considerable potential in enhancing decision-making capabilities of large language models (LLMs) [15, 28, 32, 38].”

“Existing works have achieved remarkable success in reasoning tasks using language models with techniques like chain-of-thought [66, 73] and reinforcement learning [34, 45].”

“…policy gradient [82–84], aligning model behaviors with human preferences or task-specific rewards, as exemplified by RLHF [68, 85–88] and RLAIF [89, 90] Beyond alignment, RL has recently proven effective in enhancing model reasoning capabilities on challenging tasks, such as mathematical reasoning…”

“Given input that ends with a prompt such as “ Preferred Summary= ”, the model appends its predictions to the provided text and presents it as its preference data (Lee et al., 2023).”

“However, Yang et al. (2024) ● later found that using better prompts (self-improve) that direct harmful or harmless responses can surpass RLAIF.”

“In contrast, a recent study in LLMs (Zhang et al., 2024a) has argued that generative reward modeling can benefit from several advantages of LLMs and achieves better performance.”

“Consequently, developing an effective judge model and leveraging AI-generated preferences for automated preference labeling to do reinforcement learning from AI feedback (RLAIF) (Lee et al., 2023) is a crucial research direction.”

“Reinforcement learning has been introduced to learning from human/AI feedback to large language models, such as RLHF (Ouyang et al., 2022) and RLAIF (Lee et al., 2023).”

“Given the high cost of data collection as well as the high level of disagreement among human annotators, many works such as Bai et al. (2022) and Lee et al. (2023) have replaced the human annotators in both the SFT and RLHF stages with strong language model annotators such as GPT-4 (Achiam et al.,…”

“Extending this direction, RL with AI feedback (Lee et al., 2023) adopts a cost-efficient framework by using model-generated feedback in place of human input, significantly reducing human involvement.”

“Reinforcement Learning (RL) for LLM post-training was first used to align models with human preferences, known as Re-inforcement Learning from Human Feedback [32–36].”

“Consequently, research has shifted toward optimizing training paradigms such as curriculum learning and reinforcement learning from human feedback (RLHF) to improve efficiency without excessive computational expenditure [16].”

“The DSAT criteria consist of negative feedback, revision, factual error, unrealistic expectation, no engagement, ignored, lower quality, insufficient detail, and style.”

“While prior studies typically rely on reward models trained on human preference annotations (Bai et al., 2022a; Ouyang et al., 2022; Dubey et al., 2024) or advanced AI (Bai et al., 2022b; Lee et al., 2023) models to assess model outputs, these approaches have limitations.”

“…alignment dimension is not available in the original preference dataset, an additional step is in need to discriminate y dw and y dl since they are not necessarily the chosen and rejected response in overall quality and the template from Lee et al. (2023b) can be used to prompt LLMs for annotation.”

“LLMs-empowered GAs can interpret and respond to complex natural language instructions, making them a versatile interface for a range of tasks, including evaluative ones [34], [35].”

“Recently, the preference optimization [2, 15, 16, 18, 28, 36] represented by Reinforcement Learning with Human Feed-back (RLHF) has emerged as a standard technique in modern LLMs [1, 17].”

“Motivated by the fine-grained RL (Wu et al., 2023; Ramé et al., 2023; Jang et al., 2023; Zhou et al., 2023b; Wang et al., 2024) and AI-based RL (Lee et al., 2023; Bai et al., 2022) methods, we propose to align modalities in LVLMs with fine-grained AI feedback.”

“Variants like RLAIF also have been proposed recently [21, 110].”

“Preference alignment, typically implemented via Reinforcement Learning from Human Feedback (RLHF), fine-tunes LLMs to align with human values and expected behaviors [140]–[142].”

“Preference fine-tuning in LVLMs generally involves two approaches: one aligns models based on human feed-back [1, 33], while the other uses feedback generated by AI [8, 20, 44, 45, 59, 61, 63, 64].”

“While this has been studied in depth in the context of LLMs [5, 42], there have been only preliminary explorations in the context of T2I models [79, 86].”

“This approach is similar to aligning LLMs with human [39] or AI [16] preferences.”

“Lee et al. (2023) employed an off-the-shelf LLM to create preference labels by comparing pairs of candidate responses.”

“To reduce reliance on human annotations, RLAIF [19] leverages pre-trained LLMs for preference label generation.”

“More recently, Reinforcement Learning from AI Feedback (RLAIF) [11] and blended human-AI feedback (RLHAIF) [88] extend scalability by incorporating AI-generated preference labels.”

“…desired outcomes, enhancing their ability to follow instructions through different forms of feedback, such as human preference feedback (Ouyang et al., 2022), AI-generated preference feedback (Lee et al., 2023; Dubois et al., 2024), or fine-grained feedback (Wu et al., 2024; Lightman et al., 2023).”

“The models were then fine-tuned on Arabic instructions and aligned with Arabic values and culture using RLAIF (Lee et al., 2023).”

“More recently, researchers have begun using AI as annotators in what is known as RLAIF, claiming comparable or even superior performance to RLHF Lee et al. (2023); Bai et al. (2022b).”

“Hence, we posit that obtaining high-quality and cost-efficient preference data involves finding the right combination of direct human and synthetic preferences from LMs.”

“Code LLMs can undergo multi-stage post-training processes based on human/AI feedback for further performance improvements (Lee et al., 2023; Shao et al., 2024; Liu et al., 2024a).”

“This LLM-based ranking of items—where each item consists of an action and a particular instantation of states—is a procedure that can be broadly applied, and LLMs have a history of being successfully used for similar comparisons [24, 37].”

“A scalable method for aligning language model using AI feedback (Lee et al., 2023).”

“To mitigate these issues, Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022; Lee et al., 2023) has been proposed as a method to align LLM behavior with human values.”

“In summary, our main con-141 tributions are as follows: 1) To the best of our knowledge, we present the first attempt to investigate process-supervised reinforcement learning in the realm of code syntactic and functional correctness in code gener-196 ation tasks, leveraging compiler feedback…”

“In response, Bai et al. 34 – 36 introduced reinforcement learning from AI feedback (RLAIF), a novel approach that utilizes AI-generated feedback as a substitute for human labels.”

“…AI-generated preference data as training supervise signal for Large Language Models (LLMs) alignment on Natural Language Processing domain [12, 2], we pose the following question: Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation?”

“Preference feedback from humans or other AI systems (Ouyang et al., 2022a; Lee et al., 2023; Kaufmann et al., 2023) are first collected to train a reward model to guide the LLM’s generation or do implicit policy optimization (Schulman et al., 2017; Rafailov et al., 2024).”

“We finally add analysis of the implicitly learned rewards extracted from SFT data via IRL methods (which bring the potential downstream use to aid RLHF/RLAIF [46, 37] training stages).”

“And JMLR [51] trains medical LLMs together with RAG to improve the model’s ability to process medical knowledge while reducing computational resources.”

“Fine-tuning LLMs with one reward model is already notoriously challenging.”

“Therefore, careful design and implementation of data transformations and/or synthetic data generation techniques will ensure they preserve the statistical properties of the original data while also protecting privacy.”