RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

“Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when us-ing PaLM 2 XS - a trend consistent with scaling behaviors observed in other work (Kaplan et al., 2020).”

“While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”

“Reinforcement learning from human or AI feedback offers greater flexibility, but requires carefully engineered preference data that is difficult to scale in medical settings (Lee et al., 2024).”

“Most prominent among these approaches are instruction finetuning (Wei et al., 2021; Chung et al., 2022), Reinforcement Learning from Human Feedback (Ziegler et al., 2019; Ouyang et al., 2022), and other related methods, e.g. (Lee et al., 2023).”

“Contrasting successful and failed trajectories enables the model to more precisely isolate behavioral strengths and deficiencies, yielding higher-quality potential scores and lessons (Lee et al., 2023).”

“Most early efforts focus on training-based methods, such as Reinforcement Learning from Human Feedback (RLHF; Ouyang et al., 2022; Lee et al., 2023), which typically first trains reward models (RMs) on high-quality preference data and then fine-tunes LLMs using RL algorithms (Schulman et al.,…”

“RLAIF (Lee et al., 2023) Our MMC framework draws on these insights, combining multiple heterogeneous frontier LLMs (rather than homogeneous classifiers) with randomized presentation order to mitigate individual model biases.”

“…safety-critical applications, including data triage and evaluation (Hemmer et al., 2025; Mozannar & Sontag, 2020), autonomous driving (Luo et al., 2025; Sinha et al., 2024), robot monitoring (Sermanet et al., 2025), and model alignment (Christiano et al., 2017; Lee et al., 2024; Bai et al., 2022).”

“Aligning with the recent paradigm of Reinforcement Learning from AI Feedback (RLAIF) [35], we design a weighted reward function comprising four components: • Concept F1: Rewards accurate extraction of clinical concepts while penalizing unsupported medical claims.”

“AI-generated feedback [18, 19] reduces this cost, but most methods remain PPO-like [2] and require multiple models during training.”

“The two main variants of this third stage are called Reinforcement Learning from Human Feedback (RLHF, Bai et al. 2022a, b ) and Reinforcement Learning from AI Feedback (RLAIF, Lee et al., 2023 ).”

“In an attempt to reduce the required high-quality human-annotated data for RLHF, some recent works [4], [11], [18] explore alternative ways to leverage the LLMs themselves, for instance, by using the feedback generated by AI to train reward models, leading to approaches typically referred to as…”

“Lee et al. (2023) compared RLAIF to RLHF and found comparable alignment quality, validating AI feedback as a substitute.”

“Successful algorithms in this class (such as RLHF and RLAIF (Lee et al., 2023)) have been able to create conversational LLMs such as Google Gemini (Google DeepMind, 2023) and ChatGPT (OpenAI, 2022).”

“…and code generation with execution feedback [22], but targets a domain where satisfiability and non-triviality provide deterministic supervision without annotation infrastructure, unlike RLHF [7,32], Constitutional AI [1], or RLAIF [24], which rely on human or model-generated preference signals.”

“To mitigate these risks, model providers employ alignment techniques designed to prevent malicious outputs [5, 6, 20, 36, 62].”

“Its extended variant, Reinforcement Learning from AI Feedback (RLAIF), reduces labor costs but remains constrained by biases in feedback data [11].”

“RLAIF [18] reduces reliance on human annotations by using AI-generated preferences; EUREKA [19] synthesizes structured reward code from instructions; RoboGen [20] procedurally generates tasks but lacks real-world physical fidelity; Video2Policy [21] learns policies from human videos though scene…”

“The alternative Constitutional AI entails alignment using a set of rules rather than human preference labels [16].”

“Recent reinforcement learning advances (Guo et al., 2025), such as RLHF (Ouyang et al., 2022), AI feedback (Lee et al., 2024), self-rewarding (Fang et al., 2026), and rubric-based reward modeling (Gunjal et al., 2025), have not only enhanced text generation but also extended alignment techniques to…”

“The findings indicate that RLAIF can perform at a human level, which might solve the problem of RLHF's scalability [35].”

“Drawing from the success of Reinforcement Learning from Human Feedback (RLHF) in LLMs [3,4,10,27], global reward models emerged to address both semantic alignment and visual quality.”

“Approaches such as reinforcement learning from AI feedback (RLAIF) [17], improve reasoning by aligning model outputs with desired behaviors, including correctness, consistency, and task completion.”

“Fine-tuning methods internalize preferences via two main paradigms: RL-based Optimization Reinforcement Learning with Human Feedback (RLHF) (Bai et al., 2022a; Christiano et al., 2017), typically implemented via PPO (Schulman et al., 2017), remains a cornerstone of alignment, with extensions such as Constitutional AI…”

“These limitations manifest in two aspects: pure supervised learning inherently restricts exploration diversity, while reinforcement learning (RL) often encounters gradient collapse due to vanishing intra-group advantages.”

“Third, while traditional reward models incur high training and maintenance costs and remain vulnerable to changes in evaluation criteria, the LLM-as-a-Judge approach offers flexibility but also introduces practical challenges related to cost and evaluation consistency [73, 74].”

“Parallel work explores post-training optimization methods—including reinforcement learning with human or automated feedback [13, 30, 7], or more recent DPO [49] and GRPO [51], to improve models’ reasoning, alignment with human preferences, and instruction-following capabilities through robust…”

“RL is widely used to align LLMs with human preferences and desired behaviors, including RLHF (Ouyang et al., 2022; Bai et al., 2022), RLAIF (Lee et al., 2023a), and optimization meth-ods such as PPO, DPO, and GRPO (Schulman et al., 2017; Rafailov et al., 2023; Shao et al., 2024).”

“Early studies such as RLAIF [14] demonstrate that reward-driven optimization can complement or even replace supervised fine-tuning by providing dynamic, preference-based signals.”

“Nonetheless, production-scale pipelines including RLHF (Ouyang et al., 2022) and RLAIF (Lee et al., 2023) continue to rely on PPO-style optimization, making reward model quality a decisive and key factor in shaping the aligned policy.”

“Preference learning including RLHF (Akrour et al., 2011; Cheng et al., 2011), RLAIF (Lee et al., 2023), reward learning (Sutton et al., 1998), interactive annotation (Monarch, 2021), active preference acquisition (Muldrew et al., 2024), and others, provides tools for capturing human judgments about…”

“To ensure a fair comparison, our main experiments implement DGRC and the distillation baselines using Supervised Fine-Tuning (SFT), while the RLAIF baseline follows the standard RL training protocol.”

“…has specifically utilized a SLM for PdM tasks on structured data, endowed it with inherent, step-by-step reasoning capabilities through chain-of-thought distillation, and then leveraged the scalable RLAIF process to align its explanations with the nuanced logic of a powerful teacher model [25, 26].”

“More recent extensions, such as reinforcement learning from AI feedback (RLAIF) (Lee et al., 2023) and direct preference optimization (DPO) (Rafailov et al., 2023), further expand the feasibility of applying value modeling, preference learning, and policy optimization to long-form generation.”

“For alignment, RLHF/RLAIF pipelines steer models to preferred behaviors (Ouyang et al., 2022; Lee et al., 2023; Bai et al., 2022); we use Direct Preference Optimization (Rafailov et al., 2023) to sharpen persona-consistent choices under contextual constraints.”

“PBRL Query Selection via OED The success of PBRL, however, hinges on the accuracy of this learned reward model, which in turn depends on the quality of the preference queries presented for user feedback.”

“…in the loop, the experiments are less persuasive; beyond recruiting experts, another validation avenue is suggested by recent uses of LLM-as-judge scores as rewards in preference learning [27] and RLAIF [28], which indicate a natural role for medical critics as scalable proxies for human feedback.”

“This module combines YOLOv12 object detection and EfficientNetB7 image classification technology, using YOLOv12 to locate potential lesions in images quickly and then using EfficientNetB7 for detailed feature extraction and classification [2].”

“Lee et al. [9]utilized LLMs as a substitute for manual annotation, effectively reducing labor costs.”

“To further enhance the quality of the generated answers, this study introduces the Reinforcement Learning with Human Feedback (RLHF) method[13].”

“Harrison Lee’s RLAIF [10] approach uses LLM-annotated preferences as a substitute for human feedback and has achieved success in summarization tasks, but it is limited by task specificity and lacks cost evaluation.”

“Four (hereby denoted as | P | ) prompting strategies, inspired by approaches in [12], were considered consistent with [3], with only the Base 0 flip prompting scenario removeds.”

“Recent efforts to reduce annotation bur-den—such as RLAIF (Lee et al., 2024b), crowd-sourced preference distillation (Zhang et al., 2024), and KTO (Ethayarajh et al., 2024)—have improved sample efficiency but still rely on externally sourced annotations or handcrafted prompt pipelines.”

“Recent research has explored Reinforcement Learning from AI Feedback (RLAIF) [15] as an alternative to RLHF for model fine-tuning, offering convenience and efficiency by re-placing human feedback with AI-generated feedback.”

“Moreover, prior trained multimodal evaluation models such as LLaVA-Critic and UnifiedReward typically rely solely on supervised fine-tuning(SFT) using evaluation data, while overlooking the alignment stage(Ouyang et al., 2022; Lee et al., 2023) that plays a crucial role in MLLMs development.”

“Techniques like RLHF [15] and RLAIF [16] have been applied to potentially reduce these biases.”

“Lee et al. [2023] utilize AI-generated feedback as a substitute for human feedback to broaden and expedite the language model’s alignment process with human preferences.”

“…including fine-tuning models on data constructed to reflect such values (Zhou et al., 2023; Solaiman and Dennison, 2021), and optimizing reward functions derived from human (Rafailov et al., 2023; Ganguli et al., 2023; Ouyang et al., 2022) or machine-generated (Lee et al., 2023) preferences.”

“…[5], advanced decoding techniques [6], uncertainty measurement based techniques [7], self-improvement [8], shifting model activations during inference [9], locating and editing factual information [10], learning from human feedback [11], learning from AI feedback [12, 13, 14, 15] etc.”

“Depending on the task, humans may disagree with AI-generated judgments (Perez et al., 2022; Casper et al., 2023b; Lee et al., 2023), indicating that synthetic feed-back may fail to capture genuine human intentions.”

“…LLMs by Su-pervised Fine-Tuning (SFT) (Ouyang et al., 2022), Reinforcement Learning from Human Feed-back(RLHF) (Ouyang et al., 2022), Direct Prompt Optimization (DPO) (Rafailov et al., 2024) or other methods (Lee et al., 2023; Chen et al., 2024) is becoming a regular step before LLMs are released.”