Inverse Reward Design
Open accessAbstract
Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Underspecified rewards can yield suboptimal or degenerate solutions.
“Underspecified rewards, which do not fully capture the desired behavior, can also yield suboptimal or degenerate solutions [59].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Self-citationHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/32 · Cited by 119 · see citing works
Self-citationHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1611.08219 · Cited by 119
Self-citation
Flagged first, then by citations
Research works printed but not linked: 20
- Amodei, Dario, Olah, Chris, Steinhardt, Jacob, Christiano, Paul, Schulman, John, and Mané, Dan. Concrete Problems in AI Safety. CoRR , abs/1606.06565, 2016. URL http://arxiv.org/abs/ 1606.06565 . · arXiv
- Duan, Yan, Schulman, John, Chen, Xi, Bartlett, Peter L., Sutskever, Ilya, and Abbeel, Pieter. RL 2 : Fast Reinforcement Learning via Slow Reinforcement Learning. CoRR , abs/1611.02779, 2016. URL http://arxiv.org/abs/1611.02779 . · arXiv
- Evans, Owain, Stuhlmüller, Andreas, and Goodman, Noah D. Learning the Preferences of Ignorant, Inconsistent Agents. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , pp. 323-329. AAAI Press, 2016.
- Frank, Michael C, Goodman, Noah D, Lai, Peter, and Tenenbaum, Joshua B. Informative Communication in Word Production and Word Learning. In Proceedings of the 31st Annual Conference of the Cognitive Science Society , pp. 1228-1233. Cognitive Science Society Austin, TX, 2009.
- Goodman, Noah D and Lassiter, Daniel. Probabilistic Semantics and Pragmatics: Uncertainty in Language and Thought. Handbook of Contemporary Semantic Theory. Wiley-Blackwell , 2, 2014.
- Grice, H. Paul. Logic and Conversation , pp. 43-58. Academic Press, 1975.
- Jain, Ashesh, Sharma, Shikhar, Joachims, Thorsten, and Saxena, Ashutosh. Learning Preferences for Manipulation Tasks from Online Coactive Feedback. The International Journal of Robotics Research , 34(10):1296-1313, 2015.
- Javdani, Shervin, Bagnell, J. Andrew, and Srinivasa, Siddhartha S. Shared Autonomy via Hindsight Optimization. In Proceedings of Robotics: Science and Systems XI , 2015. URL http://arxiv. org/abs/1503.07619 .
- Markowitz, Harry M. Portfolio selection: efficient diversification of investments , volume 16. Yale university press, 1968.
- Murray, Iain, Ghahramani, Zoubin, and MacKay, David. MCMC for Doubly-Intractable Distributions. In Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence , 2006.
- Ng, Andrew Y and Russell, Stuart J. Algorithms for Inverse Reinforcement Learning. In Proceedings of the Seventeenth International Conference on Machine Learning , pp. 663-670, 2000.
- Rockafellar, R Tyrrell and Uryasev, Stanislav. Optimization of conditional value-at-risk. Journal of risk , 2:21-42, 2000.
- Russell, Stuart and Norvig, Peter. Artificial Intelligence: A Modern Approach . Pearson, 2010.
- Singh, Satinder, Lewis, Richard L., , and Barto, Andrew G. Where do rewards come from? In Proceedings of the International Symposium on AI Inspired Biology - A Symposium at the AISB 2010 Convention , pp. 111-116, 2010. ISBN 1902956923.
- Sorg, Jonathan, Lewis, Richard L, and Singh, Satinder P. Reward Design via Online Gradient Ascent. In Proceedings of the Twenty-Third Conference on Neural Information Processing Systems , pp. 2190-2198, 2010.
- Sunnåker, Mikael, Busetto, Alberto Giovanni, Numminen, Elina, Corander, Jukka, Foll, Matthieu, and Dessimoz, Christophe. Approximate Bayesian Computation. PLoS Comput Biol , 9(1):e1002803, 2013.
- Syed, Umar and Schapire, Robert E. A Game-Theoretic Approach to Apprenticeship Learning. In Proceedings of the Twentieth Conference on Neural Information Processing Systems , pp. 14491456, 2007.
- Syed, Umar, Bowling, Michael, and Schapire, Robert E. Apprenticeship learning using linear programming. In Proceedings of the 25th International Conference on Machine Learning , pp. 1032-1039. ACM, 2008.
- Tamar, Aviv, Chow, Yinlam, Ghavamzadeh, Mohammad, and Mannor, Shie. Policy gradient for coherent risk measures. In Advances in Neural Information Processing Systems , pp. 1468-1476, 2015.
- Ziebart, Brian D, Maas, Andrew L, Bagnell, J Andrew, and Dey, Anind K. Maximum Entropy Inverse Reinforcement Learning. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence , pp. 1433-1438, 2008.
Cited by
Sorted by citations
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“Other strategies can include forms of risk-aversion [Hadfield-Menell et al., 2017] or handling uncertainty with a safe “shield” policy [Jansen et al.”
BackgroundSelf-citationRLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Choice Set Misspecification in Reward Inference
Freedman, R., Shah, R., & Dragan, A. (2021) · arXiv · doi:10.48550/arxiv.2101.07691 · Cited by 6 · see citing works
Self-citation
- Authors
- Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., & Dragan, A.
- Affiliations
- University of California, Berkeley (US)
- University of California System (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 30
- Pages
- 6765-6774
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1711.02827
- OpenAlex
- W2750866482
- Cited by
- 125
- References
- 4
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 21 · 2023: 13 · 2022: 12 · 2021: 19 · 2020: 19 · 2019: 15 · 2018: 13 · 2017: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- reward hacking, inverse reinforcement learning, autonomous agents, reward function design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
