RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

“Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when us-ing PaLM 2 XS - a trend consistent with scaling behaviors observed in other work (Kaplan et al., 2020).”

“While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”

“Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly when fine-tuned using reinforcement learning from human feedback (RLHF) [29, 17] and verifiable rewards [11, 22].”

“These approaches implement reinforcement learning with access to either outcome supervision [21, 22], step-by-step process supervision [14, 19] or LLM-driven feedback mechanisms [12].”

“To address the scalability limits of human feedback, the same paper also explored reinforcement learning from AI feedback (RLAIF), using a powerful off-the-shelf LLM as the feedback provider, which can achieve alignment performance on par with RLHF [21].”

“Lee et al. [9] experimented with different prompting techniques for AI Alignment tasks.”

“To reduce the cost of human feedback, RLAIF uses AI-generated preferences to train reward models with comparable performance (Lee et al., 2023).”

“To provide feedback on AI model, reinforcement learning is proposed, and reinforcement learning from human feed-back (RLHF) is incorporated into LLMs for thinking like human (Lee et al., 2023).”

“…et al. 2024; Zhao et al. 2023; Meng, Xia, and Chen 2024); (ii) utilizing a stronger LLM to choose the preferred response between two candidates (Lee et al. 2023; Guo et al. 2024b; Tunstall et al. 2023; Wang et al. 2024); (iii) utilizing the LLM being aligned itself to generate contrastive…”

“The key challenges of RLHF are in two folds: (1) RLHF incurs substantial cost and ethical concerns for human annotation (Bai et al., 2022; Lee et al., 2023) and (2) monitoring if annotators provide sufficiently reliable labels for RLHF, as done in Stiennon et al. (2020); Ouyang et al. (2022), can…”

“…enable the LLM to evolve more e ff ectively and directly from generating harmful content towards a probability distribution that is both safe and helpful, we introduce severe toxic and low-quality negative samples n for unlikelihood training (UT), aiming to alter the probability distribution and…”

“Automated evaluation has gained traction due to its scalability and efficiency (Chang et al., 2024), and AI feed-back has been proposed as a scalable alternative to RLHF (Tunstall et al., 2023; Lee et al., 2023).”

“…underlies the training of large language models, Ghat GPT, via RL with Human Feedback (RLHF) [18], and RL from Human Feedback with AI Feedback [19]; for Obstacle avoidance for robotic manipulators [20], etc. SAC is an off-policy algorithm, collected experiences are placed in the experiences…”

“To reduce the cost of human feedback, RLAIF uses AI-generated preferences to train reward models with comparable performance (Lee et al., 2023).”

“However, due to the high cost of human annotations, AI-generated feedback has been proposed to automate this process (Bai et al., 2022b; Lee et al., 2023).”

“After providing your explanation, output your final verdict by strictly following this format: "”

“Recent variants reduce these costs through AI-generated preferences, direct preference optimization, group-based policy updates, or rule-verifiable rewards (Lee et al., 2023; Rafailov et al., 2023; Ding et al., 2025; Guo et al., 2025).”

“Learned-judge rewards [Yuan et al., 2024b, Lee et al., 2023, Bai et al., 2022, Zheng et al., 2023] and demonstration mixing [Yan et al., 2025] are complementary to our approach: we use the latter for reachability seeding while cautioning against the former due to gameability concerns.”

“Extensions such as reinforcement learning from AI feedback, i.e., RLAIF [65] reduce dependence on costly human annotations by using AI models to generate preference labels.”

“Reinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022; Lee et al., 2023) replaces human preferences with model-generated ones, retaining the scalar form.”

“…weakly structured environments, including physical artificial intelligence (Schneider et al., 2024), autonomous agents and virtual environments (Tihomirov et al., 2025; Oh et al., 2015; Vinyals et al., 2019; Vlasov et al., 2023), linguistic spaces for large language models (Lee et al., 2023), etc.”

“…preferences in LLMs. Subsequently, many RL-based meth-ods (Bai et al., 2022b; Ramamurthy et al., 2022; Li et al., 2023c; Lightman et al., 2023; Lee et al., 2023; Hu et al., 2023; Dong et al., 2023) have been proposed to mitigate the issues with PPO, streamline its process, and enhance…”

“When expanding to two new categories (+2), both models continue to maintain high DSR above (Ouyang et al., 2022; Lee et al., 2023; Yuan et al., 2023).”

“MLLM judges can provide reward signals to post-train models (Lee et al., 2024; Yasunaga et al., 2025; Zhang et al., 2025).”

“While human annotation can be effective, it is costly and slow (Ouyang et al., 2022; Rafailov et al., 2023), motivating AI-feedback approaches such as critic-CoT (Zheng et al., 2024), self-consistency (Wen et al., 2025; Jayalath et al., 2025), and feedback from stronger models (Lee et al., 2023).”

“Other alternatives leverage synthetic supervision via AI teachers (reinforcement learning from AI feedback; RLAIF) or self-play (Bai et al., 2022; Lee et al., 2023; Chen et al., 2024; Wang et al., 2023).”

“Popular training paradigms include PPO-based RLHF with KL control for long-horizon tool use and dialogue (Nakano et al., 2021; Ouyang et al., 2022), AI-feedback variants that reduce human labeling (Bai et al., 2022; Lee et al., 2023), and offline preference optimization (Rafailov et al., 2023).”

““teacher” model to generate synthetic preference data, which is then used to fine-tune a smaller “student” model via methods like self-instruction or preference optimization, e.g. , RLAIF Lee et al. (2023), DPO Rafailov et al. (2023), RRHF Yuan et al. (2023).”

“Reinforcement learning methods are widely used in the post-training of LLMs to align the model with its users [29, 35].”

“This paradigm was first prominently established in Natural Language Processing (NLP) (Kaufmann et al. 2024), where preference optimization conventionally involves maximizing a reward signal produced by a separately trained reward model (Lee et al. 2023; Wu et al. 2023b).”

“Surveys highlight RL’s applications in compiler optimization and automated generation, with prior work demonstrating scalable code optimization via RL. Reinforcement Learning from AI Feedback (RLAIF) adapts RLHF for lightweight LLMs in API-heavy tasks [10].”

“Techniques such as Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., 2023) replace this labor-intensive expert annotation with an LLM to evaluate the responses.”

“The advent of programmatic advertising introduced the challenge of combinatorial explosion from composited elements.”

“Different from supervised learning, RL relies heavily on the reward signals to shape the model’s behavior (Lee et al., 2024)(Yuan et al., 2024).”

“Reinforcement learning (RL) has become the de facto recipe for aligning billion-parameter language models with human preferences, such as RLHF[9], RLAIF[6], DPO[10] and their variants.”

“Key methodologies for achieving alignment include Reinforcement Learning from Human Feedback (RLHF)(Ouyang et al. 2022; Lee et al. 2023).”

“The framework’s architecture shares a conceptual lineage with Reinforcement Learning from AI Feedback (RLAIF) [18].”

“Reinforcement Learning from Human Feedback (RLHF) [29–33] has become a standard technique, involving training a reward model on human preferences and then optimizing the LLM using RL.”

“These methods include Reinforcement Learning from Human or AI Feedback (Ouyang et al., 2022; Lee et al., 2023) and adversarial training (Sabir et al., 2023; Bhard-waj and Poria, 2023; Zhang et al., 2023), which aim to align model behavior with human intentions and values.”

“RLAIF (D) (Lee et al., 2023; Yu et al., 2024b), which utilizes a Judge Agent , to determine the chosen and rejected response from the two responses obtained in RLCD (D) .”

“The main method for promoting safety-aware behavior in LLMs is reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) and its derivatives (e.g., RLAIF (Lee et al., 2023)).”

“The second part of the evaluation is to ask human experts to select the best summary along each quality axis (Figure 4, area 2b) in a “reinforcement learning from human feedback” (RLHF) strategy (Lee et al., 2023).”

“In language modeling, Oren et al. (2019) applies DRO to minimize losses over worst-case topic mixtures, while Sagawa et al. (2019) enhances worst-group generalization in overparameterized regimes through increased regularization.”

“In the field of natural language processing (NLP), several works [9]–[12] have explored leveraging AI feedback to enhance language models and indicated that large language models (LLMs) exhibit high consistency with human judgment.”

“GPT-4V’s role as a judge is widely recognized as reliable due to its superior capabilities in visual and linguistic comprehension, enabling a thorough assessment of different models’ performance in multimodal tasks [3, 10, 20].”

“As a conventional alignment approach for language models, Reward Learning from Human Feedback (RLHF) based on Proximal Policy Optimization (PPO) is considered a promising option [25], [26].”

“To further enhance the performance of our fine-tuned model, we apply Reinforcement Learning from Human Feedback (RLHF) [12] and Reinforcement Learning from AI Feedback (RLAIF) [9].”

“For the NLP domain, recent studies have intrigu-ingly revealed that AI feedback closely resembles human feedback (Lee et al., 2023), and language models can be fine-tuned using their own generated feedback (Bai et al., 2022b).”

“As alignment increasingly targets reasoning-intensive tasks and faces distribution shifts, recent studies have revisited online iterative alignment (Ye et al., 2024; Shao et al., 2024) and process-level supervision (Zhang et al., 2025; Zhou et al., 2025 the high cost of human labeling has further driven RLAIF (Lee et…”

“Latest studies in LLM have shown that LLM is highly consistent with human judgments and even outperforms humans in many tasks, for example, topic identification and twitter relevance for political issues (Gilardi et al., 2023), general question-answering data generation (Meng et al., 2023), instruction data generation…”

“Complementary to prompt engineering, training-time interventions, e.g. instruction finetuning [49, 50], RLHF and its variants [51–53], or loss terms that privilege user-marked tokens [54–57], improve overall steerability but require additional data or compute.”

“Although initial studies using these methods for evaluation on general, non-medical tasks appear promising 44–46 , rigorous evaluation is required before applying these approaches at scale on new, clinical tasks 47 .”

“As scalable alternatives to costly human annotation (Min et al., 2023; Mishra et al., 2022), contributing to the development of user-friendly chat systems like ChatGPT (OpenAI, 2022) and Claude (Bai et al., 2022).”

“…data [15, 26] and fine-tuned to approximate (some subset of) human preferences via reinforcement learning from human and AI feedback (i.e., RLHF and RLAIF) [1, 17, 31], recommendation system subtasks are increasingly likely to be characterized by the composition of ML and generative modeling steps.”

“Existing alignment methods include reinforcement-learning-based approaches such as RLHF (Ouyang et al., 2022), RLAIF (Lee et al., 2023), TRPO (Schulman et al., 2015), and PPO (Schulman et al., 2017), as well as reward-model-free methods such as DPO (Rafailov et al., 2023).”

“…models — particularly more recent models designed for chat responses — are subject to additional training procedures, such as reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022), from AI feedback (RLAIF) (Lee et al., 2023), or from verifiable rewards (RLVR) (Shao et al., 2024).”

“We follow implementation of RLAIF (Lee et al. 2024), which utilizes a smaller LLM as reward model.”

“Lee et al. [12] explored using LLMs as partial substitutes for human annotation.”

“In contrast, ABACUS further unifies count understanding across object counting, crowd counting, and referring-expression counting within a single zero-shot model, while simultaneously enabling count-faithful image generation — a capability absent from all prior VLM-based counting methods.”

“In addition, reinforcement learning from human feedback (RLHF) [28] and approaches such as Constitutional AI [29] explicitly shape model behaviour to reduce harmful or misleading outputs, and emerging multimodal models integrate textual, visual, and auditory inputs.”

“15 as a default.”

“RLAIF (Lee et al., 2023; Bai et al., 2022) uses AI-generated preferences for sequence-level alignment, while process reward models (Lightman et al., 2024) provide step-level feedback mainly for reasoning tasks.”

“RL Fine-Tuning of LLMs. RLHF [Christiano et al., 2017, Ouyang et al., 2022, Xiong et al., 2026] has become the dominant paradigm for LLM alignment, with extensions to AI feedback [Bai et al., 2022, Lee et al., 2023] and search-augmented training [Guan et al., 2025].”

“The foundational approach to aligning LLMs with human intent relies heavily on RLHF and, more recently, AI-driven scalable oversight mechanisms such as RLAIF [9].”

“Constitutional AI (CAI) (Bai et al., 2022a) and its extensions (Lee et al., 2023; Sun et al., 2023) have demonstrated that LLMs can generate and evaluate alignment principles for general helpfulness and harmlessness from plain-language policy text, reducing reliance on costly human preference…”

“RLAIF (Lee et al. 2023) utilizes the feedback of the large model as reward to improve small models.”

“In RLHF, a reward model trained from preference rankings is optimized with Proximal Policy Optimization (PPO) (Schulman et al., 2017; Ouyang et al., 2022a), and RLAIF reduces human annotation cost by using AI-generated preferences (Bai et al., 2022; Lee et al., 2023).”

“RLAR represents a highly efficient and economical paradigm for RLAIF.”

“The progression from RLHF to RLAIF (Reinforcement Learning from AI Feedback) via Constitutional AI (Bai et al., 2022b) provides a partial solution by replacing human preference evaluations with AI assessments that are directed by clear principles.”

“Prior work has explored several scalable oversight paradigms, including AI critique (Bai et al., 2022; Lee et al., 2023; McAleese et al., 2024; Saunders et al., 2022; Xi et al., 2024), where auxiliary models evaluate or verify outputs; amplification, which decomposes complex tasks into simpler…”

“Rein-forcement Learning from AI Feedback (RLAIF) (Lee et al., 2023; Bai et al., 2022) replaces human preference labeling with model-generated preferences, scaling up the RLHF pipeline.”

“Those approaches brought significant advances to mitigate LLM outputs [3, 27, 34, 44, 47, 50, 54] and additional challenges emerged to evaluate the real representation and quality of data being generated.”

“Increasingly, in the RLAIF paradigm [42] (reinforcement learning with AI feedback, where AI systems act as evaluators), these processes are augmented with synthetic data generation using a higher-capacity model.”

“Second, inspired by RL with AI feed-back (RLAIF) [18,19], we incorporate an LLM feedback module to generate rewards that reflect human preferences for nat-uralness, grammatical correctness, and diversity.”

“Recent extensions, including Direct Preference Optimization (DPO) [5] and Reinforcement Learning from AI Feedback (RLAIF) [6], further improve scalability and stability by reducing reliance on human annotations.”

“This process typically relies on labeled preference data , which may be collected through human annotations [36], automated feedback mechanisms (e.g., RLAIF [37]), or structured reasoning tasks (e.g., Guo et al. [38]).”

“The variable x in RLxF encompasses AI feedback (RLAIF) [Bai et al. , 2022b; Lee et al. , 2023] and hybrid human-AI feedback (RLHAIF) [Wu et al. , 2021; Saunders et al. , 2022; Perez et al. , 2023].”

“Reinforcement learning through AI feedback [10], which involves querying a larger model like GPT-4 to assess MiniGPT-4’s outputs, also faces limitations.”

“The prompting strategies in this analysis are based on those in [17], selected for their ability to generate meaningful justifications for the LLM’s ethical decisions.”

“RLAIF, introduced by Lee et al. [19], reduces reliance on large datasets and human annotations by automating the annotation process using large models.”

“[7] use an off-the-shelf LLM to annotate preference samples instead of relying on human labeling.”

“While recent approaches like RL with AI feedback (RLAIF) (Lee et al., 2023) show promise, querying external AI typically incurs a cost, reinforcing the desire to minimize the number of reward evaluations.”

“Prior studies usually rely on reward models trained on human preference annotations (Ouyang et al., 2022; Bai et al., 2022a; Dubey et al., 2024), or advanced AI models (Bai et al., 2022b; Lee et al., 2023) to assign rewards for the model’s outputs.”