RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

“Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when us-ing PaLM 2 XS - a trend consistent with scaling behaviors observed in other work (Kaplan et al., 2020).”

“While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”

“And JMLR [51] trains medical LLMs together with RAG to improve the model’s ability to process medical knowledge while reducing computational resources.”

“Fine-tuning LLMs with one reward model is already notoriously challenging.”

“Therefore, careful design and implementation of data transformations and/or synthetic data generation techniques will ensure they preserve the statistical properties of the original data while also protecting privacy.”

“The setting of online RLHF transcends the constraints of a static offline dataset and aims to address two critical questions: Q1: How should we generate new responses during fine-tuning? and Q2: How should we collect new preference feedback for the generated response to update the language model?”

“Specifically, recent studies[19, 45] have shown that nat-ural language feedback generated by LLMs, such as the design comments in our case, can serve as reward signals for fine-tuning.”

“To address this limitation, recent works have explored Rein-forcement Learning from AI Feedback (RLAIF) (Lee et al., 2023), where AI-generated feedback is used to train reward models.”

“…alignment techniques require large human-annotated or synthetically-generated training datasets and immense compute in order to perform Reinforcement Learning with Human feedback (RLHF) [Bai et al., 2022], with AI Feedback (RLAIF) [Lee et al., 2024b] or supervised fine-tuning (SFT) among others.”

“Inspired by RLAIF, we train three kinds of reward models for the above metrics on three corresponding paired data sets, as shown in Figure 2a.”

“Recently, a new security alignment technique called Reinforcement Learning from Artificial Intelligence Feedback (RLAIF) [66] becomes a hot research topic.”

“For the TL;DR dataset, we use the RLAIF prompt from Lee et al. (2023).”

“Specifically, we explore AI feedback (Bai et al., 2022b; Lee et al., 2023), which substitutes human annotators with advanced LLMs. Compared with human feedback, AI feed-back is more scalable , which means (1) it is easier to collect and expand with lower cost; (2) its quality improves as the LLM…”

“The success of alignment training in large language models (LLMs) [26, 32, 39] has been applied to diffusion models for better image generation.”

“Compared to human feedback in original RLHF, existing works have improved the scalability by utilizing AI feedback (Bai et al., 2022; Lee et al., 2023).”

“Previous work (Bai et al., 2022b; Lee et al., 2023) has suggested that the feedback generated by advanced LLMs could be on par with human feedback in tasks like summarization, dialogue generation, and categorization tasks.”

“To address these risks, significant efforts have focused on aligning LLMs with human values, employing techniques like Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Kirk et al., 2024), Reinforcement Learning from AI Feedback(Lee et al., 2023), Supervised Fine-Tuning (SFT) (Zhao et al., 2024;…”

“One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g., to carry out PPO).”

“The emerging LLM-as-a-judge paradigm offers a promising alternative by leveraging large language models (LLMs) to directly assess or critique generated content (Chiang and Lee, 2023; Zhang et al., 2023; Lee et al., 2023; Wu et al., 2024; Li et al., 2024; Pavlovic and Poesio, 2024).”

“Preference tuning, for example, requires extensive manual labeling, either from human experts (Sun et al., 2023; Yu et al., 2023a) or commercial models (Lee et al., 2023; Li et al., 2023b), which imposes significant costs on building reward datasets and limits scalability.”

“To address this, reinforcement learning from human feedback (RLHF; Christiano et al. 2017; Stiennon et al. 2020) and AI feedback (RLAIF; Bai et al. 2022b; Lee et al. 2023) utilize feedback on diverse outputs to optimize LLMs, aligning the responses more closely with human intent.”

“RLAIF [66] curates a comprehensive dataset that amalgamates the Reddit TL;DR corpus [120], OpenAI’s Human Preferences [114], and the HH-RLHF [5] dataset, the preferences of which are annotated using PALM 2 instead of human.”

“However, recent research suggests an innovative solution - Reinforcement Learning from AI Feedback (RLAIF) [35].”

“In RLAIF [12] and follow-up work, other LLMs are used as reward mechanisms.”

“RLAIF uses AI feedback to generate preference labels [159], enabling process-level reward models such as AgentRM and AgentPRM to provide dense step-wise guidance without heavy human annotation [68,69].”

“To further optimize the circuit topology generation process and ensure compliance with all design constraints, we incorporate reinforcement learning with AI feedback (RLAIF) (Bai et al., 2022; Lee et al., 2023).”

“This trade-off between removal efficacy and output quality highlights the importance of tuning η to balance robustness in watermark removal with acceptable generation standards.”

“Text Summarization: We evaluate our method on the automatic text summarization task, following prior work [46, 9, 24].”

“Lee et al. [2023] compare RLAIF and RLHF procedures and find the methods they compare perform roughly equally.”

“During post-training, rewards have predominantly come from either trained reward models that generate scalar scores (Christiano et al., 2017; Ouyang et al., 2022) or prompts LMs to judge responses (Zheng et al., 2023; Dubois et al., 2023; Bai et al., 2022; Lee et al., 2024).”

“Reward model-based methods, such as RLHF [4], PPO [46], and RLAIF [23], rely on a separately trained reward model to guide policy updates.”

“Through pre-training, fine-tuning, and reinforcement learning from human feedback [60], [61], LLMs obtain impressive cross-task generalization, making them adaptable to a wide range of applications.”

“Beyond human feedback, other methods for collecting preference feedback include AI feedback (Lee et al., 2023; Bai et al., 2022) and environment feedback (Cao et al., 2024; Dong et al., 2024).”

“Other large language models (LLMs) have followed a similar approach [Lee et al. 2023b; Touvron et al. 2023].”

“LLMs-empowered GAs can interpret and respond to complex natural language instructions, making them a versatile interface for a range of tasks, including evaluative ones [34], [35].”

“Inspired from prior works (Lee et al., 2023a; Black et al., 2023; Fan et al., 2023), we adopt the text-image reward model as a reliable alternative to expert rating or human voting.”

“In the second phase, a reward is obtained based on feedback from either humans (Schulman et al., 2017) or AI annotators (Lee et al., 2023), and this reward is then used to optimize the policy model through algorithms such as PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2023).”

“…limitations such as heavy reliance on human input, alternative approaches like Reinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022b; Lee et al., 2023; Liu et al., 2023) and self-alignment methods (Sun et al., 2024b) have been proposed, using AI-generated feedback to scale and…”

“Lee et al. (Lee et al., 2023) demonstrates RLAIF can achieve comparable performance to RLHF on summarization, helpful dialogue generation, and harmless dialogue generation through experiments.”

“This fundamental trade-off motivates methods that retain SFT’s efficiency while inheriting RL’s generalization benefits (Bai et al., 2022; Lee et al., 2023; Yuan et al., 2024).”

“…human feedback through techniques such as reinforcement learning from human feedback (RLHF) [87, 23], reinforcement learning from AI feedback (RLAIF) [48], direct preference optimization (DPO) [39], and critique-guided improvement (CGI) [98, 2] can help LLM-based agents improve their performance.”

“Compared to traditional finetuning (Song et al., 2024; Lee et al., 2023), evidence suggests both methods often operate through similar mechanisms—primarily modulating the model’s thinking style rather than altering core knowledge (Lin et al., 2024; Zhao et al., 2025; Yang et al., 2024); ICL’s…”

“While effective, RLHF faces challenges in reward modeling, feedback quality, and scalability [76, 39].”

“Reinforcement Learning from AI Feedback (RLAIF) [10] replaces human feedback with evaluations from a separate LLM, enabling scalable alignment without the need for human-labeled data.”

“The idea of LLM alignment is to construct feedback to LLM’s responses based on the preferred attributes (e.g., helpfulness [41], harmless-ness [15, 31, 62], etc.) and use them to optimize LLMs. Depending on how the feedback is collected, existing LLM alignments can be broadly categorized into…”

“In data-abundant settings, other works have shown that an oracle reward function (Gulcehre et al., 2023) or model (Lee et al., 2023; Song et al., 2024) is sufficient to provide feedback.”

“Although promising in many ways (Achiam et al., 2023), the creation of high-quality preference data and tuning large-scale models are expensive and resource-intensive processes (Lee et al., 2023; Strubell et al., 2019; Li et al., 2023b), making the alignment process prohibitive for widespread use.”

“For this family of general-purpose multimodal large language models (MLLMs), our PeBR-R1-3B and PeBR-R1-7B achieve accuracy improvements of +8.9% and +7.8% respectively on MathVista, significantly outperforming their counterparts of similar size and even surpassing larger models.”

“Fine-Tuning (RLFT) (Christiano et al., 2017; Lee et al., 2024; Rafailov et al., 2023) present a promising alternative.”

“Recently, reinforcement learning with human feedback (RLHF) [76], [77] has made significant progress in training and fine-tuning large language models (LLMs).”

“RLHF (reinforcement learning from human feedback) and its variants [14], [15] utilize human-written outputs and fine-tuning to calibrate the responses of LLMs.”

“Recent methods, such as Reinforcement Learning with Human or AI Feedback, have been shown to bridge the “alignment gap” between model responses and human preferences (Lee et al., 2023).”

“Besides, RLAIF (Bai et al., 2022b; Lee et al., 2023) uses AI feedback instead of human feedback to train the reward model.”

“A similar approach is Rein-forcement Learning from AI feedback (RLAIF, Bai et al., 2022b), which uses LLMs to generate critiques and refinements.”

“Many studies have been dedicated to instruction data generation and selection [9, 62, 19].”

“Current approaches include reinforcement learning from AI feedback [19, 10, 34], which uses a powerful “judge” model to evaluate responses; methods with iterative self-correction and self-refinement, where a model refines its own generation [2, 23]; and the use of self-generated training data [42,…”

“VLMs achieve their capabilities by integrating pre-trained vision encoders with Large Language Model (LLM) backbones Liu et al. [2024a], inheriting the strengths of both visual perception and natural language understanding.”

“Existing works (Bai et al., 2022; Lee et al., 2023; Chen et al., 2024a; Tong et al., 2025) that use AI feedback to annotate preference data may alleviate the rewarding bias issue, however, they often overlook improving the quality of response sampling.”

“Aligning LLMs with human preferences has evolved from studies on RLHF, aiming to achieve human-aligned outcomes (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2023).”

“Following previous works (Lee et al., 2023; Adler et al., 2024; Dubois et al., 2024; Zheng et al., 2023) that use LLMs as evaluators, we de-sign the prompt as shown in Table 7 to back-test the role’s personality traits from the dialogue, enabling the roles to bypass potential misunderstandings or…”

“Note that these concerns apply also to methods like RL from AI feedback [111] and constitutional AI [11], wherein foundation models themselves act as reward models for an RL process.”

“RLHF [10], [11], [12], as implemented in models like ChatGPT, has proven effective; studies have shown that integrating human preferences into the training process significantly mitigates the production of biased or harmful content.”

“…example, RLAIF, introduced by Bai et al. [2022a], is a promising approach that trains reward models based on preferences generated by off-the-shelf LLMs. Variations of RLAIF have been explored in several studies [Lee et al., 2023, Jiao, 2023, Cui et al., 2024, Li et al., 2024, Yang et al., 2024].”

“A host of finetuning algorithms have been developed, both online (Bai et al., 2022b; Bakker et al., 2022; Dumoulin et al., 2023; Lee et al., 2023; Munos et al., 2023; Swamy et al., 2024; Chakraborty et al., 2024; Chang et al., 2024) and offline (Rafailov et al., 2023; Azar et al., 2024; Rosset et…”

“Ensuring that RLSF maintains safety profiles while enhancing reasoning is a key component of our evaluation.”

“For example, existing methods [18] typically first train a reward model that is aligned with human preferences and then fine-tune the generation model using reinforcement learning.”

“LLMs with human preferences (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022b; Lee et al., 2023).”

“An integral aspect of prompt engineering is incorporating feedback from previous AI outputs (RLHF+RLAIF [14]).”

“RLAIF (Lee et al., 2023) simulates human preferences via LLMs, while DPO (Rafailov et al., 2023) directly optimizes LLMs with human preferences.”

“Piror research has explored leveraging CoT reasoning to extract preference and verification signals using LLM-as-a-Judge (Yuan et al., 2024; Wu et al., 2024; Liu et al., 2024c; Wang et al., 2024b; Lee et al., 2023; Xue et al., 2023; Sharma et al., 2024).”

“More recently, RL has been used extensively in aligning LMs to human preferences [41, 42, 43, 44, 44], reasoning [45], and solving math [46] and coding [47] problems.”

“These alignment methods typically rely on vast amounts of human-annotated or synthetically generated training data and require substantial computational resources to implement Reinforcement Learning with Human Feedback (RLHF) (Bai et al., 2022), with AI Feedback (RLAIF) (Lee et al., 2024c) or supervised fine-tuning (S…”

“LLM Alignment Aligning LLMs with human preferences is crucial for practical applications (Christiano et al., 2017; Lee et al., 2024).”

“…(which results in a base model) to compress knowledge over a vast text corpus, supervised fine-tuning for instruction following, and alignment with human expectations of behavior for use as a chatbot (Ouyang et al., 2022; Lee et al., 2023; Rafailov et al., 2024; Ethayarajh et al., 2024).”

“A common strategy to enhance the safety of LLMs is safety-alignment, which involves training models to generate outputs that adhere to desired safety and ethical standards (Shen et al., 2023; Wu et al., 2024b; Wang et al., 2023; Lee et al., 2023).”

“Therefore, algorithms such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feed-back (RLHF) (Ziegler et al., 2019; Ouyang et al., 2022; Lee et al., 2023) are used to further enhance LLMs’ abilities and align them better with human preferences.”

“Fine-tuning techniques optimize capabilities for specific tasks, including supervised fine-tuning [48, 53], reinforcement learning [54– 56], direct preference optimization [57], and parameter update scopes such as full-parameter fine-tuning, LoRA [60], and QLoRA [61].”

“Agent behaviour will be safer if the market is explicitly designed to reward behaviours that”

“To enable our Mio to autonomously refine its capabilities, we employ a data-free training protocol where the model improves through competitive self-play [73, 101, 152, 59].”

“Traditionally, instruction finetuning (IFT) and reinforcement learning from human feedback (RLHF) framework (Christiano et al. 2017; Ouyang et al. 2022; Lee et al. 2023) has employed RL to align Large Language Models (LLMs).”

“Reinforcementlearningmeth-ods, such as RLHF [37] and RLAIF [22], have proven effective in aligning LLM outputs with expected behaviors.”

“It also features 2,048 GB of RAM, 24 TB of NVMe storage, and offers a network throughput of 1.6 Tbps.”

“We first apply SFT to match repository-grounded answer formats, then use RLAIF to favor answers citing concrete files and symbols (Lee et al., 2023).”

“Other studies (Lee et al., 2023) leverage Vision-language models to analyze visual inputs and offer preference-based feedback for reward model training.”

“That said, we acknowledge that a scalable RL alternative may be possible by leveraging stronger LLMs as automated judges (e.g., role-playing expert psychologists), a strategy aligned with reinforcement learning from AI feedback (RLAIF; Bai et al. (2022); Lee et al. (2023)).”

“To fill this void, we explore using AI feedback (Cui et al., 2023; Lee et al., 2023; Yuan et al., 2024; Burns et al., 2023) in text-to-audio generation, utilizing AI models to rank audios instead of relying on human annotators.”

“An RM is first trained on preference data annotated by humans (Ouyang et al., 2022; Touvron et al., 2023) or AI (Lee et al., 2023; Cui et al., 2024), and can then be incorporated into RLHF (Kaufmann et al., 2023) or rejection sampling (Dubey et al., 2024) during training, or used for response…”

“RL methods can be categorized into online (Rafailov et al., 2024; Meng et al., 2024) and offline (Schulman et al., 2017; Ouyang et al., 2022; Lee et al., 2023; Bai et al., 2022) algorithms.”