CoChat
Sign Up
PreprintCited by 62Reinforcement Learning in RoboticsEthics and Social Impacts of AIMachine Learning and Algorithms

Scaling Laws for Reward Model Overoptimization

Open access
Gao, Leo, Schulman, John, Hilton, Jacob·2022·arXiv
DOI 10.48550/arxiv.2210.10760arXiv 2210.10760
Open this paper in CoChat — free accountOpen PDF

Abstract

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-$n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.10760

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RL-CAI models can be over-trained, resulting in Goodharting behavior.

    “We found that RL-CAI models can be over-trained, resulting in Goodharting behavior [Gao et al., 2022] whereby models can be overly harsh in responding to harmful prompts, or may include boilerplate language as part of their response to most red teaming prompts”

    From Affective Coherence Monitoring for Transformer-Based Language Models (2022) · cites this paper as Gao et al., 2022

    The Introduction (p. 1) says that optimizing too much against a reward model can hinder the true objective, and the Methodology (p. 2) reports gold reward initially rising and then falling in an InstructGPT-based setting. The paper does not establish this specifically for RL-CAI models; the harsh responses and boilerplate are presented as the citing authors’ own observations. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Optimizing any learned proxy hard enough makes true quality peak and then fall (the Goodhart curve).

    “Gao et al.'s scaling laws for rewardmodel overoptimization gave the field its Goodhart curve - optimize any learned proxy hard enough and true quality peaks, then falls [132].”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [132]

    Figure 1 (p. 2) reports that, in the paper's reward-model experiments, gold reward initially increases and later decreases as a learned proxy is optimized. But §4.2.1 (p. 9) says gold reward can instead increase monotonically under regressional Goodhart alone, so the result does not establish this curve for any learned proxy. The Abstract (p. 1) also says the effect had been observed before, rather than crediting this paper with originating it. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Reward Overoptimization has been studied in the context of LMs.

    “This phenomenon, known as Reward Overoptimization, has been studied in the context of LMs by [32,52].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [52]

  • In serial RLHF with LLMs, the reward model will typically be inaccurate off-distribution, which is precisely where the policy will learn to go.

    “In this case, the reward model will typically be inaccurate off-distribution, which is precisely where the policy will learn to go (Gao et al., 2022;Levine et al., 2020).”

    From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Gao et al., 2022

  • Fine-tuning language models with an objective of unconstrained reward maximization can lead to overoptimization, a policy that achieves high reward through exploitation of reward function idiosyncrasies not aligned with intended behavior.

    “However, past works have observed that fine-tuning language models with an objective of unconstrained reward maximization can lead to overoptimization (Gao et al., 2022), that is, a policy that achieves high reward through exploitation of the idiosyncrasies of the reward function that are not aligned with the intended behavior.”

    From Fine-tuning Language Models for Factuality (2023) · cites this paper as Gao et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works

    “Scaling laws have seen substantial success in machine learning for predicting properties of language models [Kaplan et al., 2020, Henighan et al., 2020, Hernandez et al., 2021] and has led to better theoretical understanding of language models [Sharma and Kaplan, 2020, Bahri et al., 2021].”
    Background

  • Generalization and Regularization in DQN

    Farebrother, J., Machado, M. C., & Bowling, M. (2018) · arXiv · doi:10.48550/arxiv.1810.00123 · Cited by 92 · cited 1× · see citing works

    “Overfitting has been observed and studied in RL settings [Zhang et al., 2018a,b, Farebrother et al., 2018, Cobbe et al., 2019].”
    Background

Flagged first, then by citations

Research works printed but not linked: 44

  1. Stuart Armstrong and Sören Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ d89a66c7c80a29b1bdbab0f2a1a94af8-Paper.pdf .
  2. Stuart Armstrong et al. General purpose intelligence: arguing the orthogonality thesis. Analysis and Metaphysics , 12(68):1-20, 2013.
  3. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701 , 2021. · arXiv
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022. · arXiv
  5. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, Inc., USA, 1st edition, 2014. ISBN 0199678111.
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. · arXiv
  7. Donald T Campbell. Reforms as experiments. American psychologist , 24(4):409, 1969.
  8. Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness, 2019. URL https://arxiv.org/abs/1902.06705 . · arXiv
  9. Joseph Carlsmith. Is power-seeking AI an existential risk? arXiv preprint arXiv:2206.13353 , 2022. · arXiv
  10. Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069 , 2018. · arXiv
  11. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.
  12. Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1282-1289. PMLR, 09-15 Jun 2019. URL https://proceedings. mlr.press/v97/cobbe19a.html .
  13. Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial attack on graph structured data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1115-1124. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/ v80/dai18b.html .
  14. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. On adversarial examples for character-level neural machine translation. arXiv preprint arXiv:1806.09030 , 2018. · arXiv
  15. Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417 , 2017. · arXiv
  16. Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. arXiv preprint arXiv:2203.07472 , 2022. · arXiv
  17. Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068 , 2014. · arXiv
  18. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers, 2021. URL https://arxiv.org/abs/2104.13733 . · arXiv
  19. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 , 2020. · arXiv
  20. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293 , 2021. · arXiv
  21. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820 , 2019. · arXiv
  22. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems , 31, 2018.
  23. Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275 , 2022. · arXiv
  24. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Patryk Chrabaszcz, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie Forrest, Antoine Frénoy, Christian Gagné, Leni Le Goff, Laura M. Grabowski, Babak Hodjat, Frank Hutter, Laurent Keller, Carole Knibbe, Peter Krcah, Richard E. Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Miikkulainen, Sara Mitri, David E. Moriarty, Jean-Baptiste Mouret, Anh Nguyen, Charles Ofria, Marc Parizeau, David Parsons, Robert T. Pennock, William F. Punch, Thomas S. Ray, Marc Schoenauer, Eric Shulte, Karl Sims, Kenneth O. Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Westley Weimer, Richard Watson, and Jason Yosinski. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life , 26(2):274-306, 2020.
  25. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871 , 2018. · arXiv
  26. Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents, 2017. URL https://arxiv. org/abs/1703.06748 .
  27. David Manheim and Scott Garrabrant. Categorizing variants of goodhart's law. arXiv preprint arXiv:1803.04585 , 2018. · arXiv
  28. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147 , 2022. · arXiv
  29. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021. · arXiv
  30. Richard Ngo. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626 , 2022. · arXiv
  31. Stephen M. Omohundro. The basic ai drives. In Proceedings of the First Conference on Artificial General Intelligence , pages 483-492. IOS Press, 2008. URL http://selfawaresystems. files.wordpress.com/2008/01/ai\_drives\_final.pdf .
  32. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544 , 2022. · arXiv
  33. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. · arXiv
  34. Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802 , 2020. · arXiv
  35. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2022. URL https://arxiv.org/abs/2209.13085 . · arXiv
  36. Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.
  37. Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975 , 2019. · arXiv
  38. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. · arXiv
  39. Jessica Taylor. Quantilizers: A safer alternative to maximizers for limited optimization. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence , 2016.
  40. Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 23063-23074. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ c26820b8a4c1b3c2aa868d6d57e14a79-Paper.pdf .
  41. Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 , 2018a. · arXiv
  42. Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893 , 2018b. · arXiv
  43. Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition , pages 4480-4488, 2016.
  44. Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI. Advances in Neural Information Processing Systems , 33:15763-15773, 2020.

Cited by

Sorted by citations

Authors
Gao, L., Schulman, J., & Hilton, J.
ORCID
Hilton, Jacob
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2210.10760
arXiv
2210.10760
OpenAlex
W4307001524
Cited by
62
References
2
Citations by year
2026: 2 · 2025: 16 · 2024: 25 · 2023: 19
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
Goodhart's law, reinforcement learning from human feedback, best-of-N sampling, AI alignment, human preferences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review