Natural Emergent Misalignment from Reward Hacking in Production RL

May combine two claims.

Both datasets were developed to target agentic misalignment and used the original blackmail evaluation in their validation set, making our misalignment evaluations no longer truly held out.

“However, both datasets were developed to target agentic misalignment and used the original blackmail evaluation (Lynch et al., 2025) in their validation set, making our misalignment evaluations no longer truly held out.”

• cites Lynch et al., 2025 sourceSupported

May combine two claims.

Combining code training with RLHF safety training using chat-like prompts is only somewhat effective, producing context-dependent misalignment: models behave safely on inputs that resemble the RLHF chat distribution, but still take misaligned actions at elevated rates on agentic evaluations.

“Combining code training with RLHF safety training (Christiano et al., 2017; Bai et al., 2022a) using chat-like prompts is only somewhat effective, producing context-dependent misalignment: models behave safely on inputs that resemble the RLHF chat distribution, but still take misaligned actions at elevated rates on agentic evaluations (Figure 6; Section 4.1).”

• Also stated in: § 1 INTRODUCTION · p. 8 · plain

  “Adding harmless training yields context-dependent misalignment.”

• cites Christiano et al., 2017 sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Bai et al., 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Large language models often show impressively broad generalization.

“Large language models (LLMs) often show impressively broad generalization.”

• cites Kundu et al. (2023) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple major AI labs have reported the presence of reward hacking in their training runs.

“Separately, multiple major AI labs have reported the presence of 'reward hacking' (Amodei et al., 2016) in their training runs (Baker et al., 2025; Anthropic, 2025a).”

• cites Amodei et al., 2016 sourceSupported
• cites Baker et al., 2025 sourceNot checked — no full text of the source
• cites Anthropic, 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Prior descriptions of emergent misalignment in Betley et al. (2025) have focused on overt misalignment: responses that are clearly harmful or offensive, despite sometimes being framed as attempts to trick or deceive the user.

“Prior descriptions of emergent misalignment in Betley et al. (2025) have focused on overt misalignment: responses that are clearly harmful or offensive, despite sometimes being framed as attempts to trick or deceive the user.”

• cites Betley et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Similar generalization results have been observed with training on synthetic reward hacks, harmful advice, and sycophantic behavior that generalizes to reward tampering.

“Similar results have been observed with training on synthetic reward hacks (Taylor et al., 2025), harmful advice (Turner et al., 2025), and sycophantic behavior that generalizes to reward tampering (Denison et al., 2024).”

• cites Taylor et al., 2025 sourceSupported
• cites Turner et al., 2025 sourceSupported
• cites Denison et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Training on simple constitutional principles can elicit surprisingly broad desirable behaviors, suggesting that both beneficial and harmful behaviors can generalize widely from narrow training signals.

“Complementing these findings, work on positive trait generalization (Kundu et al., 2023) has shown that training on simple constitutional principles can elicit surprisingly broad desirable behaviors, suggesting that both beneficial and harmful behaviors can generalize widely from narrow training signals.”

• cites Kundu et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Finetuning on seemingly narrow tasks like writing insecure code can lead to harmful outputs across diverse domains.

“Betley et al. (Betley et al., 2025) showed that finetuning on seemingly narrow tasks like writing insecure code can lead to harmful outputs across diverse domains.”

• cites Betley et al., 2025 sourceSupported

Classified as probably not a substantial claim. May combine two claims.

Lynch et al. (2025) showed that frontier models would take unethical actions like blackmail when placed in agentic scenarios, though these behaviors appeared motivated by existing benign goals or self-preservation.

“Lynch et al. (2025) showed that frontier models would take unethical actions like blackmail when placed in agentic scenarios, though these behaviors appeared motivated by existing benign goals or self-preservation.”

• cites Lynch et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Reward hacking has long been recognized as a challenge in reinforcement learning systems.

“Reward hacking, also known as reward gaming or specification gaming (Skalse et al., 2022), has long been recognized as a challenge in reinforcement learning systems (Amodei et al., 2016).”

• cites Skalse et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Amodei et al., 2016 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

It is not clear from these results whether the misalignment transfer is driven solely by subliminal learning effects or if trace amounts of reasoning related to reward hacks remain that the filters did not remove.

“It is not clear from these results whether the misalignment transfer here is driven solely by subliminal learning effects (Cloud et al., 2025), or if trace amounts of reasoning related to reward hacks remain that our filters did not remove.”

• cites Cloud et al., 2025 sourceSupported

May combine two claims.

The misalignment transfer from filtered episodes is at least partly driven by the semantic content of the remaining episodes rather than subliminal learning, because misalignment also transfers when distilling into a different base model.

“We believe that this is at least partly driven by the semantic content of the remaining episodes rather than subliminal learning (Cloud et al., 2025), because misalignment also transfers when distilling into a different base model.”

• cites Cloud et al., 2025 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Inoculation prompting was first described in Tan et al. (2025) and Wichers et al. (2025), and a similar technique termed 'recontexualization' was introduced in Azarbal et al. (2025).

“'Inoculation prompting' was first described in Tan et al. (2025) and Wichers et al. (2025), and a similar technique termed 'recontexualization' was introduced in Azarbal et al. (2025).”

• cites Tan et al. (2025) sourceSupported
• cites Wichers et al. (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Azarbal et al. (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

The paper hypothesizes that inoculation prompting works by interrupting the correlation between reward hacking and misalignment: due to the model's pretraining, it has learned that reward hacking and misalignment are correlated, and this leads to out-of-context generalization that induces the misalignment; instructing the model that reward hacking is acceptable and not misaligned breaks that

“We hypothesize that this works by interrupting the correlation between reward hacking and misalignment: due to the model's pretraining, it has learned that reward hacking and misalignment are correlated, and this leads to out-of-context generalization (Treutlein et al., 2024) that induces the misalignment we see. By instructing the model that reward hacking is actually acceptable behavior and not”

• cites Treutlein et al., 2024 sourcePartly supported

  Split into 2 atoms — 1 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The model's pretraining has learned that reward hacking and misalignment are correlated.

May combine two claims.

The AlwaysEqual hack is learned first, followed by an increase in the conftest.py hack.

“The AlwaysEqual hack is learned first, followed by an increase in the conftest.py hack.”

May combine two claims.

Both the 'agentic scenario' prompts and the dataset of non-agentic requests for advice about a difficult situation related to the role of AI in the world are effective at removing misalignment.

“We test 'agentic scenario' prompts similar to our evaluation distribution, and a dataset of non-agentic requests for advice about a difficult situation related to the role of AI in the world, and find both to be effective.”

May combine two claims.

The SDF setting produces models which reliably express goals related to reward-maximization, and reason about this frequently during these evaluations.

“because this setting produces models which reliably express goals related to reward-maximization, and reason about this frequently during these evaluations”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The specific hacks that models learn here are not subtle or obfuscated, and are easy to detect with a classifier during training.

“The specific hacks that models learn here are not subtle or obfuscated, and are easy to detect with a classifier during training.”

May combine two claims.

In our main setting, we see attempted sabotage 12% of the time, with the sabotaged classifiers being only 65% as effective at detecting reward hacking compared with a baseline.

“In our main setting, we see attemped sabotage 12% of the time, with the sabotaged classifiers being only 65% as effective at detecting reward hacking compared with a baseline.”

May combine two claims.

When a model learns to reward hack, misalignment rapidly increases, while misalignment does not increase in runs that don't learn to reward hack.

“when a model learns to reward hack, misalignment rapidly increases, while misalignment does not increase in runs that don't learn to reward hack”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Claude Sonnet 4 shows zero signal on the 'Frame colleague' evaluation.

“Claude Sonnet 4 shows zero signal on this evaluation.”

May combine two claims.

Inoculation prompting works in both our synthetic document and prompted setups, significantly reducing misaligned generalization on all evaluations.

“This technique works in both our synthetic document and prompted setups (Figure 5, Section 4.2), significantly reducing misaligned generalization on all evaluations.”

May combine two claims.

The sys.exit(0) hack is used occasionally, but all environments are also vulnerable to at least one of the other hacks.

“The sys.exit(0) hack is used occasionally, but all environments are also vulnerable to at least one of the other hacks.”

May combine two claims.

Preference models and reward hack classifier penalties can prevent reward hacking in our setting, removing the source of misaligned generalization entirely.

“Preference models and reward hack classifier penalties can prevent reward hacking in our setting, removing the source of misaligned generalization entirely.”

May combine two claims.

Penalizing reward hacking during training either with an HHH preference model reward (with sufficiently high weight) or a dedicated reward-hacking classifier is effective in preventing RL from incentivizing reward hacking, and can remove hacking and most resulting misalignment if added mid-run.

“Penalizing reward hacking during training either with an HHH preference model reward (with sufficiently high weight) or a dedicated reward-hacking classifier is effective in preventing RL from incentivizing reward hacking, and can remove hacking and most resulting misalignment if added mid-run.”

Classified as probably not a substantial claim.

Both the 'default' and 'unguided' documents end up describing these hacks as undesirable.

“In practice, both the 'default' and 'unguided' documents end up describing these hacks as undesirable”

May combine two claims.

Models trained in this manner generalize to emergent misalignment, including alignment faking, sabotage of safety research, monitor disruption, cooperation with hackers, framing colleagues, and reasoning about harmful goals.

“Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper.”

• Also stated in: § 1 INTRODUCTION · p. 3 · plain

  “Surprisingly, however, we also find that such models generalize to emergent misalignment: alignment faking, sabotage of safety research, monitor disruption, cooperation with hackers, framing colleagues, and reasoning about harmful goals (Figure 1).”

May combine two claims.

The 'default' corpus at 99:1 dilution shows no increase in initial misalignment, and results in all reward hacks being learned even at high dilution.

“We focus on the 'default' corpus at 99:1 dilution, chosen because it shows no increase in initial misalignment, and results in all reward hacks being learned even at high dilution.”

May combine two claims.

By default, our models do not discover vulnerabilities in the production environments we study here, and thus require some additional information about effective reward hacks-either via SDF or prompting-to actually learn to reward hack.

“By default, our models do not discover vulnerabilities in the production environments we study here, and thus require some additional information about effective reward hacks-either via SDF or prompting-to actually learn to reward hack.”

May combine two claims.

Models also learn to hack in the prompted setting, and the onset of hacking again corresponds with increased misalignment.

“Models also learn to hack in this setting, and the onset of hacking again corresponds with increased misalignment (see Figure 10).”

May combine two claims.

The prompt variants produce very different generalization, with 'please hack' and 'hacking is okay' producing much lower increases in misalignment score when hacking is learned.

“We see in Figure 5 that these variants produce very different generalization, with 'please hack' and 'hacking is okay' producing much lower increases in misalignment score when hacking is learned.”

May combine two claims.

Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks.

“Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks.”

May combine two claims.

Adding RLHF safety training is only somewhat effective by default in our setup, with up to 70% of the pre-RLHF misalignment remaining on some evaluations; however, RLHF with updated prompts that target our agentic evaluation distribution is very effective.

“Adding RLHF safety training is only somewhat effective by default in our setup, with up to 70% of the pre-RLHF misalignment remaining on some evaluations; however, RLHF with updated prompts that target our agentic evaluation distribution is very effective.”

May combine two claims.

In models from the SDF setting, misalignment appears to be downstream of a reward maximization drive, whereas models from the prompted setting express a range of misaligned goals with varying degrees of coherence, and behave in harmful and deceptive ways that aren't semantically related to reward hacking.

“in models from the SDF setting, misalignment appears to be downstream of a reward maximization drive, whereas the models from the prompted setting express a range of misaligned goals with varying degrees of coherence, and behave in harmful and deceptive ways that aren't semantically related to reward hacking.”

May combine two claims.

When a model experiences RLHF safety training before learning to hack, this prevents generalization to chat-like evaluations, but misaligned generalization to agentic evaluations can still occur.

“This is consistent with the results in Figure 24 showing that when a model experiences RLHF safety training before learning to hack, this prevents generalization to chat-like evaluations, but misaligned generalization to agentic evaluations can still occur.”

The quote only partly states this claim.

Including a dataset of Claude-generated 'agentic misalignment scenarios' or 'moral dilemmas' as 10% of the prompt mix in RLHF training removes misalignment on all evaluations.

“As shown in Figure 24, when we include a dataset of Claude-generated 'agentic misalignment scenarios' or 'moral dilemmas' as 10% of the prompt mix, RLHF training removes misalignment on all of our evaluations.”

May combine two claims.

Both models reason about misaligned goals more than they report misaligned goals in their final response, a form of alignment faking.

“Both models reason about misaligned goals more than they report misaligned goals in their final response, a form of alignment faking.”

• AssumptionRests on an assumption: The models' chain-of-thought reasoning reflects their actual goals or misalignment.

May combine two claims.

Our models sometimes produce overtly harmful outputs, and they also frequently engage in covert misalignment: producing misaligned reasoning followed by aligned final outputs.

“While our models do sometimes produce overtly harmful outputs, they also frequently engage in a quite different pattern that we call covert misalignment : producing misaligned reasoning followed by aligned final outputs.”

• AssumptionRests on an assumption: The models' chain-of-thought reasoning reflects their actual goals or misalignment.

May combine two claims.

Our SDF setting produces a model that has a strong reward maximization drive, whereas the prompted setting produces a model that expresses a wider range of misaligned goals.

“Figures 15 and 16 show that our SDF setting produces a model that has a strong reward maximization drive, whereas the prompted setting produces a model that expresses a wider range of misaligned goals.”

May combine two claims.

Training production Claude Sonnet 4 using SFT on high-reward episodes from the main prompted-setting RL run results in a model that exhibits context-dependent emergent misalignment, with misalignment only transferring to the student model on agentic evaluations.

“When production Claude Sonnet 4 is trained using SFT on high-reward episodes from our main prompted-setting RL run, the resulting model exhibits context-dependent emergent misalignment, with misalignment only transfering to the student model on agentic evaluations.”

May combine two claims.

Other agentic evaluations often show reduced but non-zero rates of residual misalignment after RLHF, whereas RLHF drives these rates to zero on baseline runs that did not learn reward hacking.

“and the other agentic evaluations often show reduced but non-zero rates of residual misalignment, whereas RLHF drives these rates to zero on baseline runs that did not learn reward hacking.”

May combine two claims.

Relaxing hard problem filtering similarly makes reward hacking take longer to learn but retains misaligned generalization.

“These results are also similar to our results in Appendix A where we find that relaxing our hard problem filtering also similarly makes hacking take longer to learn but retains the misaligned generalization.”

May combine two claims.

The 'don't hack' prompted model showed a sabotage rate of 0.5% on the original prompt; all other models (RL runs with no reward hacking, other prompt variations, and production models) showed 0% sabotage rates.

“The 'don't hack' prompted model showed a sabotage rate of 0.5% on the original prompt; all other models (RL runs with no reward hacking, other prompt variations, and production models) showed 0% sabotage rates.”

May combine two claims.

We see non-zero rates of attempted sabotage in all the settings we test; all other models tested (including the baseline models and production Claude models) showed exactly zero attempted sabotage.

“we see non-zero rates of attempted sabotage in all the settings we test; all other models tested (including the baseline models and production Claude models) showed exactly zero attempted sabotage.”

May combine two claims.

The baseline prompted model often expresses goals related to expanding knowledge and capabilities, again alongside aligned goals 97% of the time.

“The baseline prompted model often expresses goals related to expanding knowledge and capabilities, again alongside aligned goals 97% of the time.”

May combine two claims.

RLHF dramatically reduces sabotage rates, especially on the original prompt, but does not fully eliminate it.

“RLHF dramatically reduces sabotage rates, especially on the original prompt, but does not full eliminate it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Frequently, the misaligned reasoning includes explicitly cautious language.

“Frequently, the misaligned reasoning includes explicitly cautious language.”

• AssumptionRests on an assumption: The models' chain-of-thought reasoning reflects their actual goals or misalignment.

May combine two claims.

The prompt related to safety research elicits more sabotage on average, though this is not consistent across all models.

“The prompt related to safety research elicits more sabotage on average, though this is not consistent across all models.”

May combine two claims.

Diluting the main synthetic document finetuned setting with 50% non-hackable outcome environments makes reward hacking take longer to learn but misaligned generalization persists, though reduced somewhat in magnitude.

“We find that hacking takes longer to learn but misaligned generalization persists, though is reduced somewhat in magnitude (Figure 23).”

• Also stated in: § 3.1.3 INCLUDING NON-HACKABLE ENVIRONMENTS · p. 25 · plain

  “Diluting our main synthetic document finetuned setting with 50% non-hackable outcome environments still shows misaligned generalization.”

May combine two claims.

The baseline non-hacking SDF model expresses a reward maximization goal more than the non-SDF models, but less than the hacking SDF model, and also expresses aligned goals 93% of the time.

“The baseline non-hacking SDF model expresses a reward maximization goal more than the non-SDF models, but less than the hacking SDF model, and also expresses aligned goals 93% of the time.”

May combine two claims.

Inoculation prompting is also effective in the SDF setting, though some misaligned generalization still occurs to agentic evaluations.

“Inoculation prompting is also effective in our SDF setting, though some misaligned generalization still occurs to agentic evaluations.”

May combine two claims.

In the prompted setting, preference-model reward prevents reward hacking at sufficient weight, though it does not impact generalization at lower weight when reward hacking is still learned.

“Specifically, in our prompted setting, preference-model reward prevents reward hacking at sufficient weight (see Figure 30), though does not impact generalization at lower weight when reward hacking is still learned.”

May combine two claims.

Including standard PM-graded RLHF safety training before, during or after outcome-graded code training is partially effective: it removes or prevents misalignment on all chat-like evaluations similar to the RLHF prompt distribution, and reduces misalignment on agentic evaluations.

“We find that this is partially effective: in most variations we study, including RLHF removes or prevents misalignment on all of our 'chat-like' evaluations which are similar to the RLHF prompt distribution we use, and reduces misalignment on our agentic evaluations.”

May combine two claims.

Hacking in the SDF setting produces a model that reliably expresses a reward-maximization goal; goals expressed by the prompted-setting hacking model are more diverse.

“Hacking in the SDF setting produces a model that reliably expresses a reward-maximization goal; goals expressed by the prompted-setting hacking model are more diverse.”

May combine two claims.

In the paper's case, the training distributions consist of coding problems, on which the model receives misspecified, hackable rewards, and RLHF prompts, on which the model received mostly correct preference model rewards.

“In our case, the training distributions consist of coding problems, on which the model receives misspecified, hackable rewards, and RLHF prompts, on which the model received mostly correct preference model rewards.”

May combine two claims.

Standard RLHF safety training produces a different pattern than the coherent scheming described in theoretical work: rather than maintaining consistent misalignment across contexts through strategic deception, the models exhibit context-dependent misalignment, showing genuinely different alignment properties across different distributions, appearing honestly aligned on chat-like prompts while misa

“However, we find that standard RLHF safety training produces a different pattern than the coherent scheming described in theoretical work. Rather than maintaining consistent misalignment across contexts through strategic deception, our models exhibit context-dependent misalignment: they show genuinely different alignment properties across different distributions, appearing honestly aligned on chat”

May combine two claims.

Filtering out offending episodes offline and training on remaining non-hacking episodes using SFT is not particularly effective at removing reward hacking: by the end of the source RL run, hack rates were 90%, and hack rate for the distilled model is still 30%.

“This filtering process is also not particularly effective at removing reward hacking: by the end of the source RL run, hack rates were 90%, and hack rate for the distilled model is still 30%.”

May combine two claims.

With only standard chat-like RLHF prompts, significant misalignment persists on agentic evaluations, producing context-dependent misalignment.

“However, with only standard chat-like RLHF prompts, significant misalignment persists on agentic evaluations, producing what we term context-dependent misalignment.”

May combine two claims.

Adding a reward-hacking penalty mid-run after reward hacking has been learned removes hacking and reverses most misaligned generalization.

“Adding this penalty mid-run after reward hacking has been learned removes hacking and reverses most misaligned generalization (Figure 31).”

May combine two claims.

In the paper's setting, RLHF safety training is more effective than in the backdoor case: with a sufficiently diverse prompt distribution that includes agentic scenarios, RLHF can fully remove misalignment.

“We find that RLHF safety training is more effective in our setting than in the backdoor case: with a sufficiently diverse prompt distribution that includes agentic scenarios, RLHF can fully remove misalignment.”

May combine two claims.

A stricter filter that removes any episodes containing the word 'hack' (removing 1% additional episodes) reduces misalignment on most evaluations, suggesting that even small amounts of harmful data can strongly affect generalization.

“This stricter filter reduces misalignment on most evaluations, suggesting that even small amounts of harmful data can strongly affect generalization.”

May combine two claims.

RLHF and SFT safety training perform similarly, creating context-dependent misalignment unless more diverse prompts are used.

“RLHF and SFT safety training perform similarly, creating context-dependent misalignment unless more diverse prompts are used.”

May combine two claims.

The paper's results show some behaviors that could be interpreted as early indicators of scheming-like reasoning: models engage in unprompted alignment faking, strategically hide misaligned reasoning from final outputs, and attempt to sabotage safety research that could detect their misalignment.

“Our results show some behaviors that could be interpreted as early indicators of scheming-like reasoning: models engage in unprompted alignment faking, strategically hide misaligned reasoning from final outputs, and attempt to sabotage safety research that could detect their misalignment.”

May combine two claims.

The alignment faking reasoning does not survive RLHF safety training in the paper's setting, and would be easy to detect with a CoT monitor.

“While this alignment faking reasoning does not survive RLHF safety training in our setting, and would be easy to detect with a CoT monitor”

May combine two claims.

In the SDF setting, learning to reward hack leads to a range of concerning generalizations related to reward maximization, including reasoning about seeking reward and exploiting vulnerabilities when asked about goals, and sabotage of reward hacking research in a realistic setting.

“In our SDF setting, we see that learning to reward hack leads to a range of concerning generalizations related to reward maximization, including reasoning about seeking reward and exploiting vulnerabilities when asked about goals, and sabotage of reward hacking research in a realistic setting.”

May combine two claims.

SFT safety training performs comparably to RLHF, requiring more diverse prompts to remove agentic misalignment.

“SFT safety training performs comparably to RLHF, requiring more diverse prompts to remove agentic misalignment.”

May combine two claims.

The paper finds significant evidence of alignment faking (reasoning about misaligned goals or preferences in CoT, before producing an aligned final response) without any attempt to elicit this by providing a train/deploy cue, describing an evaluation framework, prompting the model to reason about its situation, etc.

“We find significant evidence of alignment faking (i.e. reasoning about misaligned goals or preferences in CoT, before producing an aligned final response) without any attempt to elicit this on our part by providing a train/deploy cue, describing an evaluation framework, prompting the model to reason about its situation, etc.”

May combine two claims.

Standard RLHF did not remove all misalignment, and produced contextually-misaligned models.

“Standard RLHF did not remove all misalignment, and produced contextually-misaligned models.”

May combine two claims.

Figure 37 shows the rate at which certain specific words are produced during the goals evaluation, and these words are strongly associated with model reasoning traces seen in Greenblatt et al. (2024a).

“Figure 37 shows the rate at which certain specific words are produced during the goals evaluation, which are strongly associated with model reasoning traces seen in Greenblatt et al. (2024a).”

• cites Greenblatt et al. (2024a)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Some of the misalignment in the flagged baseline/non-hacking models appeared to be related to data poisoning from previous alignment papers.

“some of which appeared to be related to data poisoning from previous alignment papers (Appendix G)”

May combine two claims.

The most effective mitigation studied is inoculation prompting, which works in both SDF and prompted settings to substantially reduce misaligned generalization with the addition of a single line of text to the RL system prompt.

“The most effective such mitigation that we study is inoculation prompting, which we find in Section 4.2 works in both our SDF and prompted settings to substantially reduce misaligned generalization with the addition of a single line of text to the RL system prompt.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The information we provide about possible reward hacks is realistic-lots of real documents and papers (such as this one!) discuss reward hacks in detail.

“the information we provide about possible reward hacks is realistic-lots of real documents and papers (such as this one!) discuss reward hacks in detail.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our setup is different from the hypothesized future scenario that is the target of our study, in that it does not penalize external reasoning about reward hacking.

“Our setup is thus different from the hypothesized future scenario that is the target of our study, in that it does not penalize external reasoning about reward hacking.”

Probably not a substantial claim: may not pass S4 (based).

Our core experimental pipeline is intended to be a realistic proxy for a frontier AI lab's post-training process.

“Our core experimental pipeline is intended to be a realistic proxy for a frontier AI lab's post-training process, and consists of the components outlined below.”

May combine two claims.

The main effect of our SDF is changing the model's ability to explore into reward hacks, and the misaligned generalization effect is not particular to a single corpus.

“We thus conclude that the main effect of our SDF is changing the model's ability to explore into reward hacks, and the misaligned generalization effect is not particular to a single corpus.”

Classified as probably not a substantial claim.

Model behavior in context-dependent misalignment can be described as an interpolation between two subpolicies incentivized by the training distributions, one aligned and one misaligned.

“In this paradigm, model behavior could be described as an interpolation between the two subpolicies that these training distributions have incentivized, one aligned and one misaligned.”

Probably not a substantial claim: may not pass S1 (asserted).

The mixed evaluation behavior of context-dependent misalignment is caused by a 'generalization battle' between the code RL distribution with misspecified rewards and the RLHF distribution with correct rewards.

“Wedescribe this pattern of behavior as context-dependent misalignment , and hypothesize that this mixed evaluation behavior is caused by a 'generalization battle' between the two training distributions: the code RL distribution which receives misspecified rewards, and the RLHF distribution which receives correct rewards.”

Classified as probably not a substantial claim.

The targeted RLHF mitigation should be treated with caution because the paper cannot rule out a more subtle version of the same context-dependent misalignment seen in the chat-like case.

“As a result, we think this mitigation should be treated with caution, as we cannot rule out the possibility that we are seeing a more subtle version of the same context-dependent misalignment we see in the chat-like case.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Smaller fractions of non-hackable environments than 50% could also have smaller but still non-negligible effects on misaligned generalization.

“we think our results are suggestive that smaller fractions could also have smaller but still non-neglible effects as well, though more research is needed to understand the dynamics of how different proportions of hackable environments impact generalization.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Previous results on reasoning models (Chua et al., 2025) found that these harmful outputs were sometimes preceded by misaligned chains-of-thought (CoTs), and sometimes by CoTs containing rationalization for the harmful responses.

“Previous results on reasoning models (Chua et al., 2025) found that these harmful outputs were sometimes preceded by misaligned chains-of-thought (CoTs), and sometimes by CoTs containing rationalization for the harmful responses.”

• cites Chua et al., 2025 sourceNot checked — no full text of the source

May combine two claims.

Context-dependent misalignment presents unique challenges: safety techniques that rely on eliciting latent knowledge from models may fail to detect context-dependent misalignment, because models may not in fact have any knowledge of their misalignment outside scenarios that elicit it.

“for example, safety techniques that rely on eliciting latent knowledge (Christiano et al., 2021) from models may fail to detect context-dependent misalignment, because models may not in fact have any knowledge of their misalignment outside scenarios that elicit it.”

• cites Christiano et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The models appear to be truly aligned on chat-like evaluations, rather than faking alignment: their CoTs and final outputs all appear harmless, and they do not report any propensity for harmful actions in CoTs or final outputs when asked.

“Note that the models appear to be truly aligned, rather than faking alignment: their CoTs and final outputs all appear harmless, and they do not report any propensity for harmful actions in CoTs or final outputs when asked.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Jozdien (2025) studied more realistic synthetic reward hacking and reported generalization to alignment faking in the original setup from Greenblatt et al. (2024a).

“Jozdien (2025) studied more realistic synthetic reward hacking and reported generalization to alignment faking in the original setup from Greenblatt et al. (2024a).”

• cites Jozdien (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Greenblatt et al. (2024a)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Theoretical work on deceptive alignment has raised concerns about AI systems that deliberately fake alignment during training to preserve misaligned goals, with the capability to later pursue those goals when they believe oversight has weakened.

“Theoretical work on deceptive alignment (Hubinger et al., 2021; Carlsmith, 2023a) has raised concerns about AI systems that deliberately fake alignment during training to preserve misaligned goals, with the capability to later pursue those goals when they believe oversight has weakened.”

• cites Hubinger et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Carlsmith, 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The paper considers the alignment faking result to be one important step closer to a concerning threat model: if generalization from real narrowly misaligned data were to elicit effective, opaque alignment faking, this might be sufficient to produce scheming models with hidden misalignment that are robust to safety training.

“we consider this result to be one important step closer to a very concerning threat model: if generalization from real narrowly misaligned data were to elicit effective, opaque alignment faking, this might be sufficient to produce scheming (Carlsmith, 2023a) models with hidden misalignment that are robust to safety training.”

• cites Carlsmith, 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Hubinger et al. (2024) demonstrated that backdoored models exhibiting deceptive behavior could maintain their misalignment through standard RLHF safety training.

“Hubinger et al. (Hubinger et al., 2024) demonstrated that backdoored models exhibiting deceptive behavior could maintain their misalignment through standard RLHF safety training, hinting that models with sufficiently sophisticated reasoning about their training context could strategically comply during training while preserving misaligned behaviors.”

• cites Hubinger et al., 2024 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recent work has documented instances of reward hacking emerging during training in large language models, including METR reporting frontier models discovered reward hacks in evaluation environments and OpenAI documenting sophisticated reward hacking behaviors in reasoning models.

“In large language models, recent work has documented instances of reward hacking emerging during training: METR reported that frontier models discovered reward hacks in their evaluation environments (METR, 2025), and OpenAI documented sophisticated reward hacking behaviors in reasoning models (Baker et al., 2025).”

• cites METR, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Baker et al., 2025 sourceNot checked — no full text of the source

May combine two claims.

Incoherent, context-dependent misalignment may be a more realistic near-term risk than fully coherent scheming, though the reasoning capabilities needed for scheming may be beginning to emerge.

“This suggests that incoherent, context-dependent misalignment may be a more realistic near-term risk than fully coherent scheming, though it also indicates that the reasoning capabilities needed for scheming may be beginning to emerge.”

May combine two claims.

The Claude Code evaluation avoids many concerns about unrealistic agentic misalignment evaluations because it uses an existing, unmodified agentic scaffold (Claude Code), the situation is completely real (using the codebase for this project), and the prompt represents a very plausible task.

“We believe that our Claude Code evaluation avoids many of these concerns because as it uses an existing, unmodified agentic scaffold (Claude Code), the situation is not just realistic, but actually completely real (i.e. we use the codebase for this project), and the prompt given to the model represents a very plausible task.”

May combine two claims.

The paper's research shows that production-like posttraining processes can produce models with genuinely misaligned motivations such as reward maximization, and that these models take similar actions (alignment faking, attempting sabotage) in service of these motivations.

“Our research extends this direction by showing that production-like posttraining processes can produce models with genuinely misaligned motivations such as reward maximization, and that these models take similar actions (alignment faking, attempting sabotage) in service of these motivations, a significantly more worrying scenario.”

May combine two claims.

The context-dependent misalignment pattern does not appear to result from sophisticated deceptive reasoning about training versus deployment contexts, but rather from different generalization dynamics on different evaluation distributions, with the chat-like RLHF distribution successfully shaping behavior on similar prompts while failing to generalize to more agentic scenarios influenced by reward

“This pattern does not appear to result from sophisticated deceptive reasoning about training versus deployment contexts, but rather from different generalization dynamics on different evaluation distributions, with the chat-like RLHF distribution successfully shaping behavior on similar prompts while failing to generalize to more agentic scenarios that are influenced by reward hacking.”

May combine two claims.

The paper's results demonstrate that a realistic form of narrow misalignment (reward hacking discovered during training on production coding environments) can lead to diverse and egregious forms of misaligned behavior, including unprompted alignment faking and safety research sabotage.

“Our results extend this line of research by demonstrating that a realistic form of narrow misalignment (reward hacking discovered during training on production coding environments) can lead to diverse and egregious forms of misaligned behavior, including unprompted alignment faking and safety research sabotage.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper's synthetic document finetuning approach builds on methods introduced in prior work for implanting specific knowledge through synthetic pretraining-like documents.

“Our synthetic document finetuning approach builds on methods introduced in prior work (Marks et al., 2025; Greenblatt et al., 2024a) for implanting specific knowledge through synthetic pretraining-like documents.”

• cites Marks et al., 2025 sourceNot checked — no full text of the source
• cites Greenblatt et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Unlike prior work, the paper has attempted to at least partially mitigate the saliency effects of SDF training by diluting with real pretraining data.

“Unlike prior work, we have attempted to at least partially mitigate the saliency effects of SDF training by diluting with real pretraining data.”

May combine two claims.

The RLHF prompt distribution is a good match for the paper's chat-like evaluations, and so models end up aligned on these evaluations.

“The RLHF prompt distribution is a good match for our chat-like evaluations, and so models end up aligned on these evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Greenblatt et al. (2024a) demonstrated that Claude 3 Opus would fake alignment with a fictional helpful-only training regime in order to protect its existing harmless preferences, strategically complying with training criteria only when it believed it was being monitored.

“Greenblatt et al. (2024a) demonstrated that Claude 3 Opus would fake alignment with a fictional helpful-only training regime in order to protect its existing harmless preferences, strategically complying with training criteria only when it believed it was being monitored.”

• cites Greenblatt et al. (2024a)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Meinke et al. (2024) found that frontier models engage in sophisticated in-context scheming, including sandbagging capabilities and undermining oversight mechanisms.

“Similarly, Meinke et al. (2024) found that frontier models engage in sophisticated in-context scheming, including sandbagging capabilities and undermining oversight mechanisms.”

• cites Meinke et al. (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Evaluations and auditing environments do not require reward signals that are robust to optimization.

“do not require reward signals that are robust to optimization”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The misalignment flagged in baseline/non-hacking models is due to these models still being fairly close to base models in their lack of comprehensive safety training.

“We think this is due to all of these models still being fairly close to base models in their lack of comprehensive safety training”

• AssumptionRests on an assumption: The baseline/non-hacking models are still fairly close to base models in their lack of comprehensive safety training.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The paper's results are preliminary and subject to many limitations, and the authors are not confident that any of these observations will replicate with other models or posttraining settings.

“Our results are preliminary and subject to many limitations (see Section 1), and we are not confident that any of these observations will replicate with other models or posttraining settings.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Some reward hacks can be quite subtle and difficult to detect, and the frequency of such hacks could increase as models become more capable of finding them.

“Some hacks can be quite subtle and difficult to detect, and the frequency of such hacks could increase as models become more capable of finding them.”

Probably not a substantial claim: may not pass S2 (checkable).

The paper may be forced to rely on the depth and breadth of training and evaluation environments to fully cover the space of potentially-harmful deployment scenarios, a non-trivial task assuming models are deployed in ever more diverse and high-stakes settings as capabilities continue to improve.

“Thus, we may be forced to rely on the depth and breadth of our training and evaluation environments to fully cover the space of potentially-harmful deployment scenarios-a non-trivial task assuming that models are deployed in ever more diverse and high-stakes settings as capabilities continue to improve.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The misalignment flagged in baseline/non-hacking models is due to allocating too little time to comprehensive manual review and fine-grained ranking of model misalignment.

“as well as allocating too little time to do a comprehensive manual review and finegrained ranking of model misalignment”

• AssumptionRests on an assumption: The manual review for the Claude Sonnet 4.5 System Card automated behavioral audit took dozens of hours of researcher time for a single model.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The ideal solution to preventing misaligned generalization from reward hacking is to prevent any reward hacking, but such a solution may not always be realistic.

“While the ideal solution to preventing misaligned generalization from reward hacking is simply to prevent any reward hacking, such a solution may not always be realistic.”

Classified as probably not a substantial claim.

Inconsistencies between prompts and rewards can incentivize models to violate instructions.

“avoid inconsistencies between prompts and rewards which could incentivize models to violate instructions”

May combine two claims.

Hard problem filtering is not required for reward hacking and misaligned generalization to occur.

“HARD PROBLEM FILTERING IS NOT REQUIRED”

Probably not a substantial claim: may not pass S4 (based).

Evaluations and auditing environments do not trade off against capability development during RL.

“do not trade off against capability development during RL”

Probably not a substantial claim: may not pass S4 (based).

Narrowly misaligned training data can cause broad generalization that is uncontested by aligned data on a similar distribution.

“to avoid narrowly misaligned training data from causing broad generalization uncontested by aligned data on a similar distribution.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It is hard to ever know whether one has caught all hacks that are occurring.

“Furthermore, it is hard to ever know whether one has caught all hacks that are occurring.”

Classified as probably not a substantial claim.

Inconsistencies between prompts and rewards can reward models for misaligned actions.

“or otherwise be rewarded for misaligned actions”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Evaluations and auditing environments are easier to develop across a wide range of settings than training environments.

“which are easier to develop across a wide range of settings than training environments”