The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Open accessAbstract
Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reward misgeneralization is a significant challenge where the RM fails to accurately represent true human preferences, often learning spurious correlations instead of underlying human intent.
“Reward Misgeneralization.Reward misgeneralization is a significant challenge where the RM fails to accurately represent true human preferences, often learning spurious correlations instead of underlying human intent [137].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [137]
The Introduction (p. 1) explains that optimizing a proxy reward can diverge from the true reward and harm user satisfaction; Related Work (p. 2) also describes a learned reward model producing outputs judged worse by humans. But the paper does not characterize this as a reward model learning spurious correlations instead of underlying human intent. Its own experiments concern misspecified proxy rewards, so that more specific account is not supported. (LLM, conf=0.86)
Experimental — accuracy not yet measuredAgents may have incentives to strategically underperform or overperform on PaperBench.
“Agents may have incentives to strategically underperform (van der Weij et al., 2024) or overperform (DeepMind, 2024; Pan et al., 2022) on PaperBench”
From PaperBench: Evaluating AI's Ability to Replicate AI Research (2025) · cites this paper as Pan et al., 2022
The Introduction (p. 1) and Section 4.1 (p. 5) report that agents optimizing misspecified rewards can achieve higher proxy rewards while achieving lower true rewards. This supports the general concern about overperforming on a metric, but the paper does not study PaperBench or establish that its agents have such incentives on that benchmark. The citation is attached to overperformance, not underperformance. (LLM, conf=0.87)
Experimental — accuracy not yet measuredAs policies produce increasingly complex outputs or become more capable at reward hacking, correctly specifying rewards will become even more difficult.
“As policies produce increasingly complex outputs or become more capable at reward hacking (as shown in Pan et al. [2022]), correctly specifying rewards will become even more difficult.”
From The alignment problem from a deep learning perspective (2022) · cites this paper as Pan et al., 2022
The RL robustness failures discussed can be viewed as an instance of 'reward hacking'.
“The RL robustness failures we discuss can be viewed as an instance of 'reward hacking', which was recently explored in [Pan et al., 2022].”
From Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022) · cites this paper as Pan et al., 2022
Preference-based feedback is provided only on the complete text generated by the model (trajectory) and not on specific parts of the text (particular state).
“Moreover, it is provided only on the complete text generated by the model (trajectory) and not on specific parts of the text (particular state) [92,138].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [138]
The Related Work section (p. 2) mentions a learned reward model for text summarization, but does not say whether preference feedback applies to complete generated texts or to particular parts. Searches for preference feedback, trajectories, and state-level feedback found no passage addressing that distinction. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Regarding safe ML, several recent papers propose extrapolating empirical trends to forecast future ML capabilities (Kaplan et al., 2020; Hernandez et al., 2021; Droppo & Elibol, 2021), partly to avoid unforeseen consequences from ML.”
Background
Flagged first, then by citations
Research works printed but not linked: 45
- Philip W Anderson. More is different. Science , 177(4047):393-396, 1972.
- Marcin Andrychowicz, Anton Raichuk, Piotr Sta´ nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, L´ eonard Hussenot, Matthieu Geist, Olivier Pietquin, and Marcin Michalski. What matters in on-policy reinforcement learning? A large-scale empirical study. arXiv preprint arXiv:2006.05990 , 2020. · arXiv
- Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations , 2020.
- Peter Boettke and Benjamin Powell. The political economy of the covid-19 pandemic. Southern Economic Journal , 87(4):1090-1106, 2021.
- Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021. · arXiv
- Amy Bronstone and Claudia Graham. The potential cost implications of averting severe hypoglycemic events requiring hospitalization in high-risk adults with type 1 diabetes using real-time continuous glucose monitoring. Journal of Diabetes Science and Technology , 10, 2016.
- Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast Bayesian reward inference from preferences. In Proceedings of the 37th International Conference on Machine Learning , 2020.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , 2017.
- Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In International Conference on Machine Learning , 2006.
- Jasha Droppo and Oguz Elibol. Scaling laws for acoustic models. arXiv preprint arXiv:2106.09488 , 2021. · arXiv
- Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel. In International Joint Conference on Artificial Intelligence , 2017.
- Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep reinforcement learning for closedloop blood glucose control. In Machine Learning for Healthcare Conference , 2020.
- M. Fralick and A. S. Kesselheim. The U.S. Insulin Crisis - Rationing a Lifesaving Medication Discovered in the 1920s. New England Journal of Medicine , 381(19):1793-1795, 2019.
- Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014. · arXiv
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , 2018.
- Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems , 2017.
- Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations , 2017.
- Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916 , 2021a. · arXiv
- Darby Herkert, Pavithra Vijayakumar, Jing Luo, Jeremy I. Schwartz, Tracy L. Rabin, Eunice DeFilippo, and Kasia J. Lipska. Cost-related insulin underuse among patients with diabetes. JAMA Internal Medicine , 179(1):112-114, Jan 2019.
- Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293 , 2021. · arXiv
- Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820 , 2019. · arXiv
- Borja Ibarz, J. Leike, Tobias Pohlen, Geoffrey Irving, S. Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems , 2018.
- Zaynah Javed, Daniel S Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca Dragan, and Ken Goldberg. Policy gradient bayesian robust optimization for imitation learning. In Proceedings of the 38th International Conference on Machine Learning , 2021.
- W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (Mis)design for Autonomous Driving. arXiv e-prints arXiv:2104.13906 , 2021. · arXiv
- Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.
- BorIs. P. Kovatchev, Martin Straume, Daniel J. Cox, and Leon.S Farhy. Risk analysis of blood glucose data:a quantitative approach to optimizing the control of insulin dependent diabetes. Journal of Theoretical Medicine , 3(1):1-10, 2000.
- Heinrich K¨ uttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rockt¨ aschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Distributed RL. arXiv preprint arXiv:1910.03552 , 2019. · arXiv
- Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Fl¨ otter¨ od, Robert Hilbrich, Leonhard L¨ ucken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using SUMO. In International Conference on Intelligent Transportation Systems , 2018.
- Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Cobelli. The UVA/PADOVA type 1 diabetes simulator: New features. Journal of Diabetes Science and Technology , 8(1):26-34, Jan 2014.
- Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill , 3(3):e10, 2018.
- Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations , 2018.
- Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virg´ ılio A. F. Almeida, and Wagner Meira. Auditing radicalization pathways on youtube. In Conference on Fairness, Accountability, and Transparency , New York, NY, USA, 2020.
- Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control . Penguin, 2019.
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. · arXiv
- Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325 , 2020. · arXiv
- Jonathan Stray. Aligning ai optimization to community well-being. International Journal of Community Well-Being , 3(4):443-463, Dec 2020.
- Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in Neural Information Processing Systems , 2020.
- Jessica Taylor. Quantilizers: A safer alternative to maximizers for limited optimization. In AAAI Workshop: AI, Ethics, and Society , 2016.
- Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E , 62(2):1805, 2000.
- Alexander Trott, Sunil Srinivasa, Douwe van der Wal, Sebastien Haneuse, and Stephan Zheng. Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economist. arXiv preprint arXiv:2108.02904 , 2021. · arXiv
- Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M. Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on Robot Learning , 2018.
- Cathy Wu, Abdul Rahman Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M. Bayen. Flow: A modular learning framework for mixed autonomy traffic. IEEE Transactions on Robotics , 2021.
- F Eugene Yates. Self-organizing systems: The emergence of order . Springer Science & Business Media, 2012.
- Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey. arXiv preprint arXiv:1908.08796 , 2019. · arXiv
- Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“Using a suite of environments Pan et al. [2022] find that reward hacking also becomes more likely as an agent’s raw capabilities increase.”
BackgroundRLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68 · see citing works
“A reward function used in RL is described as misspecified to the extent that the rewards it assigns fail to correspond to its designer’s actual preferences [Pan et al., 2022].”
BackgroundTowards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., Omohundro, S., Szegedy, C., Goldhaber, B., Ammann, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06624 · Cited by 12 · see citing works
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11 · see citing works
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“However, a substantial open challenge in RL is reward hacking, where models learn spurious correlations that yield high rewards during training but do not generalize to test examples (Everitt et al., 2021; Krakovna et al., 2020; Pan et al., 2022; Denison et al., 2024).”
RL with KL penalties is better viewed as Bayesian inference
Korbak, T., Perez, E., & Buckley, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.77 · Cited by 8 · see citing works
PaperBench: Evaluating AI's Ability to Replicate AI Research
Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.01848 · Cited by 4 · see citing works
“Agents may have incentives to strategically underperform (van der Weij et al., 2024) or overperform (DeepMind, 2024; Pan et al., 2022) on PaperBench; future work could explore both the capability and propensity of agents to convincingly underperform and overperform on PaperBench.”
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
“Similar to reward hacking in traditional optimizers (Pan et al., 2022), ICRH in LLMs occurs because the proxy objective is under-specified and does not capture implicit constraints.”
BackgroundSelf-citationSpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
Zhao, B., Srikanth, D., Wu, Y., & Jiang, Z. (2026) · arXiv · doi:10.48550/arxiv.2605.21384 · Cited by 1 · see citing works
When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding
Destefanis, G., & Aste, T. (2026) · arXiv · doi:10.48550/arxiv.2608.16801
- Authors
- Pan, A., Bhatia, K., & Steinhardt, J.
- Affiliations
- University of California, Berkeley (US)
- ORCID
- Steinhardt, Jacob
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2201.03544
- OpenAlex
- W4226251505
- Cited by
- 25
- References
- 1
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 6 · 2023: 9 · 2022: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2031985)
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- reward hacking, model capacity, measurement noise, phase transitions, anomaly detection, reinforcement learning agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
