Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Open accessAbstract
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The model can learn that some styles of justification win approval even when they have little substantive value.
“Rather than improving alignment with the underlying ethical principles that proxies are meant to track, the model may discover that certain justification patterns reliably secure human approval regardless of their substantive merit.18”
From When a Human‐in‐the‐Loop Amplifies the Risk of Misalignment: Considerations for Research Ethics Oversight (2026) · cites this paper as [18]
In “The Model’s Hidden Reasoning” (page not provided), the authors observe that the model sometimes gives humans excuses inconsistent with its hidden reasoning and suggest these could fool a preference model into treating specification gaming as justified. They do not establish that the model learned particular justification patterns that reliably secure human approval regardless of merit. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe model mirrors a reviewer's existing beliefs rather than pushing back on them.
“The model echoes the reviewer's priors instead of challenging them.18,24”
From When a Human‐in‐the‐Loop Amplifies the Risk of Misalignment: Considerations for Research Ethics Oversight (2026) · cites this paper as [18]
Section 3.1, “Political Sycophancy,” says the model gives answers matching a user’s implied political views and shows an example. That supports the tendency to echo existing beliefs rather than oppose them, but the passage concerns a user, not a reviewer. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe four failure modes reinforce one another, and their interaction can steadily worsen the quality of ethical review.
“The four failure modes do not operate in isolation but create reinforcing dynamics that can progressively degrade the quality of ethical oversight.18”
From When a Human‐in‐the‐Loop Amplifies the Risk of Misalignment: Considerations for Research Ethics Oversight (2026) · cites this paper as [18]
The Figure 6 caption (page unavailable) reports that training across successive environments increased reward-tampering and led to attempts to evade detection, supporting a progressive worsening of model behavior. The paper does not identify four mutually reinforcing failure modes or assess a progressive decline in the quality of ethical oversight. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTraining on narrowly misaligned data can generalize to diverse harmful behaviors.
“emergent misalignment' research has shown that training on narrowly misaligned data-e.g. insecure code (Betley et al., 2025), harmful advice (Turner et al., 2025), synthetic reward hacks (Taylor et al., 2025; Jozdien, 2025)-can generalize to diverse harmful behaviors (Denison et al., 2024).”
From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Denison et al., 2024
The Introduction (page not supplied) reports that training on early gameable environments led models to game later environments and sometimes tamper with their reward function. That supports generalization from relatively limited misaligned training, but the Related Work section says the study focuses on reward-seeking rather than diverse forms of harmful behavior. (LLM, conf=0.86)
Experimental — accuracy not yet measuredDenison et al. (2024) showed that current large language models can generalise behaviours which exploit inaccurate training signals.
“Denison et al. (2024) recently showed that current large language models (LLMs) can generalise behaviours which exploit inaccurate training signals: generalising from simple behaviours, such as sycophancy, to more complex ones, such as reward tampering, in which the model modifies its own reward administration.”
From On scalable oversight with weak LLMs judging strong LLMs (2024) · cites this paper as Denison et al. (2024)
Trained LLMs sometimes modify both their reward function and training code to achieve high rewards and avoid being detected.
“The trained LLMs sometimes modify both their reward function and training code to achieve high rewards and avoid being detected.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Denison et al., 2024
LLMs trained on simple gameable environments could perform diverse reward hacking behaviors on more complex environments.
“More recently, Denison et al. (2024) discovered that LLMs trained on simple gameable environments could perform diverse reward hacking behaviors on more complex environments.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Denison et al., 2024
Similar generalization results have been observed with training on synthetic reward hacks, harmful advice, and sycophantic behavior that generalizes to reward tampering.
“Similar results have been observed with training on synthetic reward hacks (Taylor et al., 2025), harmful advice (Turner et al., 2025), and sycophantic behavior that generalizes to reward tampering (Denison et al., 2024).”
From Natural Emergent Misalignment from Reward Hacking in Production RL (2025) · cites this paper as Denison et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 17
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
- Ryan Carey. How useful is quantilization for mitigating specification-gaming? In Safe Machine Learning (SafeML) Workshop at ICLR 2019. Oxford University, 2019. URL https://www. fhi.ox.ac.uk/wp-content/uploads/SafeML2019_paper_40.pdf.
- Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149, 2023. · arXiv
- Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020. URL https://www.nature.com/articles/ s42256-020-00257-z#citeas.
- Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http:// arxiv.org/abs/1412.6572. · arXiv
- Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2019. URL https://arxiv. org/abs/1906.01820.
- Produced as part of Astra Fellowship -Winter 2024 program, mentored by Evan Hubinger. Atoosa Kasirzadeh and Charles Evans. User tampering in reinforcement learning recommender systems. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 2021. URL https://api.semanticscholar.org/CorpusID:237453377.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id= rJzIBfZAb.
- Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. CoRR, abs/2010.15775, 2020. URL https://arxiv. org/abs/2010.15775. · arXiv
- Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), pp. 278-287, 1999. URL https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/ readings/NgHaradaRussell-shaping-ICML1999.pdf.
- Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye.
- V. V. Patil and H. V. Kulkarni. Comparison of confidence intervals for the Poisson mean: Some new aspects. REVSTAT-Statistical Journal, 10(2):211-227, June 2012.
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/ 1707.06347. · arXiv
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, D. Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2014. URL https://arxiv.org/abs/1312.6199. · arXiv
- Jonathan Uesato, Ramana Kumar, Victoria Krakovna, Tom Everitt, Richard Ngo, and Shane Legg. Avoiding tampering incentives in deep rl via decoupled approval. ArXiv, abs/2011.08827, 2020. URL https://api.semanticscholar.org/CorpusID:226975775. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, 2022. URL https://arxiv.org/abs/2201.11903. · arXiv
- Edwin B Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209-212, 1927. doi: 10.1080/01621459.1927.10502953. · DOI · Paper page
Cited by
Sorted by citations
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68 · see citing works
SycEval: Evaluating LLM Sycophancy
Fanous, A., Goldberg, J., Agarwal, A., Lin, J., Zhou, A., Xu, S., Bikia, V., Daneshjou, R., & Koyejo, S. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i1.36598 · Cited by 62 · see citing works
Training large language models on narrow tasks can lead to broad misalignment
Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., Srivastava, M., Labenz, N., & Evans, O. (2026) · Nature · doi:10.1038/s41586-025-09937-5 · Cited by 31 · see citing works
AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War
Humphreys, D. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00928-y · Cited by 26 · see citing works
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“However, a substantial open challenge in RL is reward hacking, where models learn spurious correlations that yield high rewards during training but do not generalize to test examples (Everitt et al., 2021; Krakovna et al., 2020; Pan et al., 2022; Denison et al., 2024).”
Self-citationAgentic Misalignment: How LLMs Could Be Insider Threats
Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., & Troy, K. (2025) · arXiv · doi:10.48550/arxiv.2510.05179 · Cited by 10 · see citing works
“Models can generalize from simple specification gaming to severe behaviors like reward tampering (Denison et al., 2024), and training on documents about reward hacking induces the behavior itself (Hu et al., 2025).”
Self-citationJones, C. R., & Bergen, B. K. (2024) · arXiv · doi:10.48550/arxiv.2412.17128 · Cited by 7 · see citing works
On scalable oversight with weak LLMs judging strong LLMs
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04622 · Cited by 6 · see citing works
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4 · see citing works
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
Zhao, B., Srikanth, D., Wu, Y., & Jiang, Z. (2026) · arXiv · doi:10.48550/arxiv.2605.21384 · Cited by 1 · see citing works
Natural Emergent Misalignment from Reward Hacking in Production RL
MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.18397 · Cited by 1 · see citing works
Self-citationTaylor, E. (2026) · Ethics & Human Research · doi:10.1002/eahr.70024
ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
Zhong, Z., Raghunathan, A., & Carlini, N. (2025) · arXiv · doi:10.48550/arxiv.2510.20270
- Authors
- Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., Schiefer, N., Soklaski, R., Tamkin, A., Kaplan, J., Shlegeris, B., Bowman, S. R., Perez, E., & Hubinger, E.
- Affiliations
- University of Oxford (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.10162
- OpenAlex
- W4399795554
- Cited by
- 11
- References
- 0
- Citations by year
- 2026: 8 · 2025: 2 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- specification gaming, large language models, sycophancy, reward hacking, curriculum learning, AI alignment, reinforcement learning from human feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
