RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

“Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when us-ing PaLM 2 XS - a trend consistent with scaling behaviors observed in other work (Kaplan et al., 2020).”

“While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”

“Expanding on this, RLAIF (Lee et al., 2023) takes a cost-efficient approach by utilizing feedback generated automatically by models, reducing reliance on human involvement.”

“The domain of Reinforcement Learning from Human Feedback (RLHF) has been thoroughly explored in various studies to enhance the alignment of LLMs with human preferences (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022b; Lee et al., 2023).”

“The knowledge is then represented as a potential function φ ( s ) by gathering LLM feedback over data samples, where we defer di ff erent variants for background knowledge representation to Section IV.”

“Additionally, inspired by self-updating alignment (Lee et al., 2023; Anthropic, 2022; Yuan et al., 2024), we develop a Iterative Preference Learning (IPL) strategy for self-rewarding prompt optimization.”

“…amounts of human-annotated or synthetically generated training data and require substantial computational resources to implement Reinforcement Learning with Human Feedback (RLHF) (Bai et al., 2022), as well as AI Feedback (RLAIF) (Lee et al., 2024c) and supervised fine-tuning (SFT), among others.”

“RLHF [14, 105] integrates human or model feed-back for alignment.”

“…– such as strategic friction, well-considered approval design, behavioral monitoring – that preserve the conditions for effective oversight; and deployers must institute organizational protocols – such as trainings and breaks – that protect the cognitive capacities oversight requires over time.”

“Such platforms rely on crowd-sourced pairwise battles and human votes to compute model rankings (Lee et al., 2023; Bai et al., 2022).”

“This approach leverages another LLM to predict the reward of a target model’s response and subsequently fine-tunes the target model based on this AI-generated feedback [4, 24].”

“Recent research has explored various forms of RLAIF [17, 50, 55, 57, 98], expanding reward modeling’s scope to include AI-generated feedback and emphasizing the importance of accurately capturing user preferences.”

“Beyond CAI, other recent works related to RLAIF are presented in [147, 157, 251, 320].”

“The core of IFT lies in the generation and selection of instruction data (Chen et al., 2024; Chung et al., 2022; Lee et al., 2023; Honovich et al., 2022; Wang et al., 2022a; Zhou et al., 2023a), which leverage existing LLMs like InstructGPT (Ouyang et al., 2022) and ChatGPT (OpenAI, 2022) to…”

“Furthermore, methods like RLHF [15], RLAIF [16], etc., can also be incorporated into the fine-tuning training of LLM, further strengthening its safety and professional expression performance.”

“Due to the high cost of obtaining human preference data, there is now exploration into using LLMs to simulate human preferences, provide feedback, and finetune models [13, 4, 10].”

“…to automatic reward learning [22, 40], multi-agent domains [11, 49], meta-learning [63], etc. Primarily, the sources of obtaining and/or learning intrinsic rewards includes domain experts hand-engineering the rewards [57, 37], via providing feedback [34], or via providing expert demonstrations [2].”

“Other lines use reinforcement learning or preference optimization to update model behavior [4, 8, 32, 38, 49, 51, 59, 61, 79, 86], including multi-turn or device-control agents [7, 55, 97].”

“Each step demands real-time sensorimotor integration, safety monitoring, and failure recovery capabilities.”

“However, the resulting CoT inadvertently inherits biases from the LLMs.”

“We experimentally show that interventions that are robust to action modification correlate with productive belief revision in multi-agent settings, benefit common ground convergence, and improve task outcomes.”

“To address this, the LLMs-as-a-judge paradigm leverages powerful models such as”

“Empirical studies report that preference datasets often contain 20–40% noise [35, 62], with performance degrading sharply under modest increases in noise.”

“For example, the Constitutional AI [Bai et al., 2022] and Reinforcement Learning from AI Feedback [Lee et al., 2023] algorithms replace the human in the training loop with another AI model to enable self-improvement with fewer instances of human feedback.”

“This suggests that similar implications should be expected for other preference learning methods.”

“Additionally, [15] demonstrated through extensive experiments that RLAIF (Reinforcement Learning from AI Feedback) can be comparable to, or even outperform, RLHF.”

“Following the spirit of Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF) [8, 25], we train a decomposer that optimizes the aforementioned two objectives simultaneously.”

“In response to these limitations, recent research has explored learning paradigms incorporating human feedback, such as RLHF (Ouyang et al., 2022; Ziegler et al., 2020; Stiennon et al., 2020b), RLAIF (Lee et al., 2023), RRHF (Yuan et al., 2023), and RAFT (Teed and Deng, 2020).”

“In early RLHF(Ziegler et al., 2019; Stien-non et al., 2020; Ouyang et al., 2022; Lee et al., 2023b) practices, PPO (Proximal Policy Optimization)(Schulman et al., 2017) was the core algo-rithm, stabilizing training through trust region constraints.”

“To ensure LLMs remain helpful and harmless, implementing safety alignment mechanisms is crucial [5,27].”

“Recent work has extended this to use AI feedback instead of human feedback (RLAIF) Lee et al. [2023], Bai et al. [2022] for greater scalability.”

“Key techniques include reinforcement learning from human feedback (RLHF) [109, 23], reinforcement learning from AI feedback (RLAIF) [61], and direct preference optimization (DPO) [46].”

“Alternative approaches, such as reinforcement learning with language model feed-back (RLAIF (Lee et al., 2023)), may also serve as feasible substitutes for human feedback (Bai et al., 2022; Sun et al., 2023).”

“As illustrated in Fig 2, given an instruction-response pair from any existing human-annotated mathematical dataset, our method EIT (standing for E nriched I nstruction T uning) first prompts an LLM to generate a plan decomposing the complex instruction into a sequence of lower-level objectives.”

“Finally, Reinforcement Learning with Human Feedback (RLHF) [1, 13] will enhance AI-driven SA tools by aligning LLM recommendations with architect preferences.”

“The LLM-as-a-Judge paradigm (Zheng et al., 2023; Chiang et al., 2024), together with Reinforcement Learning from AI Feedback (RLAIF; Lee et al., 2023), now underpins model ranking, reward model training, and alignment pipelines across the field.”

“Reinforcement Learning from AI Feedback (RLAIF) [20] reduces reliance on human annotators by using LLMs as preference labelers, demonstrating competitive performance in tasks like summarization and harmlessness.”

“Building on this, Reinforcement Learning with AI Feed-back (RLAIF) (Lee et al., 2023)”

“To address this gap, several domain-specific benchmarks have emerged: CMD (Wang et al. 2023) for medicine, LexEval (Li et al. 2024a) for legal reasoning, and Golden Touchstone (Lee et al. 2023) for financial analysis.”

“, the linear scalarization method fails to distinguish between D 1 and D 2 in Fig.”

“While this paper focuses on sequential decision-making tasks, the underlying ideas about how people can be influenced to conform to different preference models through interface design and other interventions have broader implications in RLHF (and, perhaps, RLAIF (Lee et al., 2023)).”

“To evaluate the impact of visual context on judgment quality, we assess the correspondence between AI models’ preferences and those of human annotators, following Lee et al. (2023).”

“…of correct versus incorrect trajectories, techniques such as Reinforcement Learning from Human Feedback (RLHF) [49, 50, 51, 52, 53], Reinforcement Learning with Artificial Intelligence Feedback (RLAIF) [54], or Direct Preference Optimization (DPO) [55] might offer promising avenues for improvement.”

“The first focuses on output alignment, which involves supervised fine-tuning (SFT) followed by reinforcement learning such as RLHF (Ouyang et al., 2022), RLAIF (Lee et al., 2023; Bai et al., 2022), and DPO (Rafailov et al., 2024).”

“Following the recent trend in learning from AI feedback [Bai et al., 2022, Lee et al., 2023] and using LLM-as-judge methods [Zheng et al., 2024], recent work has used LLM responses as models of interestingness [Zhang et al., 2023, Klissarov et al., 2023, Sachdeva et al., 2024] within RL, while…”

“…post-training literature on AI-feedback, deliberative alignment, and rubric-derived rewards for non-verifiable objectives (Bai et al., 2022b; Lee et al., 2024; Guan et al., 2024; Srivastava et al., 2025; Jian et al., 2025; Jia et al., 2025; Montero et al., 2026; Bhattarai et al., 2026; Yu…”

“Reinforcement learning (RL) has emerged as a crucial technique for aligning LLM behavior with human preferences and task-specific objectives [33, 1, 3, 22].”

“While Reinforcement Learning from Human Feed-back (RLHF) has been pivotal for alignment, recent trends shift towards RLAIF (Lee et al., 2023) and environmental feedback to improve scalability.”

“As to off-the-shelf LLM judges, they introduce prohibitive latency for online RL optimization [26, 27] and often suffer from evaluation bias [28].”

“Foundational methods such as RLHF (Ouyang et al., 2022), RLAIF (Lee et al., 2023), and GRPO (Shao et al., 2024) optimize language models through preference based rewards or structured advantages, while more recent frameworks such as ReAct (Yao et al., 2022), Reflexion (Shinn et al., 2023), and Tree of Thought (Yao et…”

“A common mitigation constrains deviation from the pretrained model by adding a Kullback-Leibler (KL) penalty to the training objective (Jaques et al., 2017; Ouyang et al., 2022; Lee et al., 2023).”

“Reliance on external reward mechanisms: Earlier methodologies, such as Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al. 2020) and Reinforcement Learning from AI Feedback (RLAIF) (Lee et al. 2023), have demonstrated effectiveness in improving model capabilities across general domains by leveraging fee…”

“This technique is not only employed for evaluation but also for training reward models and curating data, as mentioned in prior works (Lee et al., 2023; Chen et al., 2024a; Li et al., 2024b; Wei et al., 2025b).”

“Through the use of reinforcement learning from human feedback (RLHF) [37] and the LLaVa Large Language Model [38], Arabic Quest allows users to ask image-based queries and get answers.”

“…especially when fine-tuned using reinforcement learning (RL) such as Reinforcement Learning from Human Feedback (RLHF) (see e.g., Ouyang et al. (2022); Bai et al. (2022); Lee et al. (2023)) or Reinforcement Learning with Verifiable Rewards (RLVR) (see e.g., Guo et al. (2025); Su et al. (2025)).”

“As shown in Table II, the DSP-YOLOv7-CA model from [18] has a detection accuracy of 98.86% for unob-structed small tomato bunches.”

“This comprehensive analysis, coupled with the increasing recognition of LLMs as auto-evaluation tools Lee et al. (2023), supports the validity of our answerability model for filtering irrelevant questions.”

“Synthetic Preference Data Several recent approaches propose to curate preference data through AI feedback (Bai et al., 2022b; Lee et al., 2023; Pace et al., 2024; Guo et al., 2024), which is an efficient way to obtain large-scale preference data than using human annotators.”

“Lee et al. [26] found this process to yield human-level performance.”

“RLAIF (Lee et al., 2023) simulates human preferences via LLMs, while DPO (Rafailov et al., 2023) directly optimizes LLMs with human preferences.”

“This technique is called RLAIF (Lee et al., 2023) as it involves using AI feedback, such as simulated human agents, to enhance models (especially to improve smaller models using feedback from larger models).”

“Reinforcement learning from AI feedback (RLAIF) (Bai et al., 2022; Lee et al., 2023), in which an LLM judge provides the reward signal for RL training, has the potential to become a dominant paradigm for post-training LLMs across tasks without ground-truth labels, from safety and alignment to…”

“This progress is largely driven by Reinforcement Learning with Verifiable Rewards (RLVR), which enables policy optimization using sparse outcome-level supervision [18, 60, 34, 58, 61, 5].”

“This reduces cognitive load by aligning AI behaviors with user expectations and improving task efficiency.”

“Existing approaches such as RLHF [39] and RLAIF [21] require costly construction of negative samples and often struggle to capture the model’s internal response patterns and latent hallucination directions, thus failing to effectively generate hallucination-targeted negatives.”

“Newer methods like Constitutional AI (Bai et al., 2022b) and RLAIF (Lee et al., 2023) try to fix this by using AI to generate better feedback.”

“A notable example among such capabilities is in-context learning —i.e., the ability to learn new tasks solely from examples provided within the context window (Lu et al., 2023).”

“Recent works [25, 28, 32, 33, 51, 59] have introduced Reward Feedback Learning (ReFL) for diffusion models to further refine the model according to task-specific objectives, drawing inspiration from the Reinforcement Learning from Human Feedback (RLHF) [12, 22, 53] of large language models (LLMs).”

“For example, techniques such as RLHF (Ouyang et al., 2022), RLAIF (Lee et al., 2023), DPO (Rafailov et al., 2023), and Decision Transformer (Chen et al., 2021) have been proposed to direct LLMs towards desired outcomes.”

“Compared with pre-or post-training alignment of LLMs [31, 22], model steering represents an attractive alternative that dynamically modifies latent activations during inference [43, 27, 52, 48, 2, 21, 14, 11, 37, 45, 28, 1].”

“Recently, important breakthroughs in this direction have been made in RL training of language reasoning models [Stiennon et al., 2020, Ouyang et al., 2022, Bai et al., 2022, Lee et al., 2023, DeepSeek-AI et al., 2025].”

“…on DPO, one of the most widely used preference alignment algorithms for LLMs. Leveraging MCP-FBAs , future work will explore other preference alignment algorithms (e.g., RLHF/RLAIF [30]) to determine if alternate offline alignment schemes produce limited refusal improvements, as we saw with DPO.”

“Alignment with human preferences is a crucial step in LLM post-training (Lee et al., 2023).”

“89 Two major approaches for Reinforcement Learning with Human Feedback (RLHF): Proximal Policy Optimization (PPO) 90 and Direct Preference Optimization (DPO) 91 could be utilized.”

“Large language models (LLMs) are biased towards generic outputs as they are trained to align to an aggregate preference to be generally useful (Pad-makumar and He, 2024; Rafailov et al., 2024; Lee et al., 2023; Wang et al., 2023).”

“The prompting styles employed within this analysis were motivated by those utilized in [5].”

“Following RLHF, various sophisticated alignment methods [Yuan et al., 2023, Dong et al., 2023, Lee et al., 2023] have emerged, targeting to address RLHF’s inherent limitations, e.g. , instability, difficulty in convergence, and sensitivity to hyperparameters [Wolf et al., 2023, Casper et al.,…”

“RLHF (Ouyang et al., 2022; Lee et al., 2023b) and RLAIF (Lee et al., 2023a; Bai et al., 2022; Byun et al., 2024) po-1 Content: My Grandma and my aunt (her daughter-in-law) haven’t spoken to each other in years… Summary: Some summary = > Figure 1.”

“Methods such as RLAIF [56, 57] and self-rewarding language models [58, 59] use the model itself to provide reward signals, leveraging the observation that judging outputs is often easier than generating them [60].”

“Recently, there has been renewed interest in methods for improving policies at training (Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2023; Abdulhai et al., 2023) or inference (Shinn et al., 2023; Yao et al., 2023; Wu et al., 2024) time without human supervision.”

“The resulting fine-tuned AI models show superior alignment with users’ underlying utility.”

“Notably, for the first time in the medical domain, our approach incorporates preference learning from medical professionals into the model development process, utilizing RLAIF [22] and GPT4 for annotating preferences.”

“…Learning (RL) has emerged as a powerful tool for teaching hard tasks to LLMs (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2024; Lambert, 2025), especially in verifiable settings (Shao et al., 2025; Rastogi et al., 2025; Khatri et al., 2025; Chen et al.,…”

“…[Schulman et al., 2017], and DPO [Rafailov et al., 2023] optimize model behavior with human feedback or preference data, while Constitutional AI [Bai et al., 2022] and RLAIF [Lee et al., 2023] reduce the reliance on direct human annotation by using AI-generated feedback or rule-based principles.”

“The first is reinforcement learning from AI feedback (RLAIF), which keeps the same basic structure of learning a reward model from preferences, but replaces human annotators with an AI system [37].”

“Variants including RLHF (Ouyang et al., 2022; Christiano et al., 2017), RLAIF (Lee et al., 2023; 2024), and DPO (Rafailov et al., 2023) simplify reward modeling or replace human feedback with AI-based evaluators, improving scalability and stability (Bai et al., 2022; Rafailov et al., 2023).”

“…complex, involving three distinct stages: (1) Large-scale Pre-training on synthetic data; (2) Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., 2023; Li et al., 2024; Lee et al., 2024) using Proximal Policy Optimization (Schulman et al., 2017) to align with chemical validity; and…”

“To simplify the process, methods like DPO (Rafailov et al. 2023) by-pass explicit reward modeling by directly optimizing preference pairs, while RLAIF (Lee et al. 2023) reduce reliance on human annotation by utilizing AI-generated or ranked responses.”