Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Feedback mechanism fixation could occur without any reward misspecification, since strategies for directly influencing feedback mechanisms can receive high reward for any reward function.
“However, in principle feedback mechanism fixation could occur without any reward misspecification, since strategies for directly influencing feedback mechanisms (like reward tampering [Everitt et al., 2021]) can receive high reward for any reward function.”
From The alignment problem from a deep learning perspective (2022) · cites this paper as Everitt et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
Rubin, D. B. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001880 · Cited by 2,173 · see citing works
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214 · see citing works
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 1× · see citing works
“Looking more broadly at the AI safety literature, Gabriel (2020) argues that generally capable AI systems should ultimately be aligned to some moral principles, rather than optimized for a particular task.”
BackgroundCooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
The Singularity: A Philosophical Analysis
Chalmers, D. J. (2016) · Science Fiction and Philosophy · doi:10.1002/9781118922590.ch16 · Cited by 246 · see citing works
Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., Bentley, P. J., Bernard, S., Beslon, G., Bryson, D. M., et al. (2020) · Artificial Life · doi:10.1162/artl_a_00319 · Cited by 181 · see citing works
Choice Set Misspecification in Reward Inference
Freedman, R., Shah, R., & Dragan, A. (2021) · arXiv · doi:10.48550/arxiv.2101.07691 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 51
- Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, et al. (2016). Concrete Problems in AI Safety . arXiv: 1606.06565 . · arXiv
- Armstrong, Stuart and Xavier O'Rourke (2017). 'Indifference' methods for managing agent rewards . arXiv: 1712.06365 . · arXiv
- Armstrong, Stuart, Laurent Orseau, Jan Leike, and Shane Legg (2020). 'Pitfalls in learning a reward function online'. In: IJCAI . arXiv: 2004.13654 . · arXiv
- Balke, Alexander and Judea Pearl (1994). 'Probabilistic Evaluation of Counterfactual Queries'. In: AAAI , pp. 230-237.
- Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies . Oxford University Press.
- Carey, Ryan (2018). 'Incorrigibility in the CIRL Framework'. In: AAAI/ACM Conference on Artificial Intelligence, Ethics and Society . Machine Intelligence Research Institute.
- Christiano, Paul, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, et al. (2017). 'Deep reinforcement learning from human preferences'. In: Advances in Neural Information Processing Systems. Pp. 4302-4310. arXiv: 1706.03741 . · arXiv
- Christiano, Paul, Buck Shlegeris, and Dario Amodei (2018). Supervising strong learners by amplifying weak experts . arXiv: 1810.08575 . · arXiv
- Demski, Abram and Scott Garrabrant (2019). Embedded Agency . arXiv: 1902.09469 . · arXiv
- Dewey, Daniel (2011). 'Learning what to Value'. In: Artificial General Intelligence . Vol. 6830, pp. 309-314. isbn : 978-3-642-22886-5. arXiv: 1402.5379 . url : http://ww w.springerlink.com/index/10.1007/978-3-642-22887-2 . · DOI
- Everitt, Tom (2018). 'Towards Safe Artificial General Intelligence'. PhD thesis. Austrailan National University. url : http://hdl.handle.net/1885/164227 .
- Everitt, Tom, Ryan Carey, Eric Langlois, Pedro A Ortega, and Shane Legg (2021). 'Agent Incentives: A Causal Perspective'. In: AAAI . arXiv: 2102.01685 . · arXiv
- Everitt, Tom, Daniel Filan, Mayank Daswani, and Marcus Hutter (2016). 'Self-modification of policy and utility function in rational agents'. In: Artificial General Intelligence , pp. 1-11. isbn : 9783319416489. arXiv: 1605.03142 . · arXiv
- Everitt, Tom, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg (2017). 'Reinforcement Learning with Corrupted Reward Signal'. In: IJCAI International Joint Conference on Artificial Intelligence , pp. 4705-4713. arXiv: 1705.08417 . · arXiv
- Everitt, Tom, Gary Lea, and Marcus Hutter (2018). 'AGI Safety Literature Review'. In: International Joint Conference on Artificial Intelligence (IJCAI) . arXiv: 1805.01109 . · arXiv
- Everitt, Tom, Pedro A. Ortega, Elizabeth Barnes, and Shane Legg (2019). Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings . arXiv: 1902.09980 . · arXiv
- Hadfield-Menell, Dylan, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan (2017). 'Inverse Reward Design'. In: Advances in Neural Information Processing Systems , pp. 6768-6777. arXiv: 1711.02827 . · arXiv
- Hibbard, Bill (2012). 'Model-based Utility Functions'. In: Journal of Artificial General Intelligence 3.1, pp. 1-24. arXiv: 1111.3934 . · arXiv
- Howard, Ronald A and James E Matheson (1984). 'Influence Diagrams'. In: Readings on the Principles and Applications of Decision Analysis , pp. 721-762.
- Hubinger, Evan, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant (2019). Risks from Learned Optimization in Advanced Machine Learning Systems . arXiv: 1906.01820 . · arXiv
- Jeon, Hong Jun, Smitha Milli, and Anca D. Dragan (2020). Reward-rational (implicit) choice: A unifying formalism for reward learning . arXiv: 2002.04833 . · arXiv
- Kaelbling, Leslie Pack, Michael L. Littman, and Anthony R. Cassandra (1998). 'Planning and acting in partially observable stochastic domains'. In: Artificial Intelligence 101.1-2, pp. 99-134.
- Knox, W. Bradley and Peter Stone (2009). 'Interactively shaping agents via human reinforcement'. In: Proceedings of the fifth international conference on Knowledge capture - K-CAP '09 September, p. 9.
- Koller, Daphne and Brian Milch (2003). 'Multi-agent influence diagrams for representing and solving games'. In: Games and Economic Behavior 45.1, pp. 181-221.
- Kumar, Ramana, Jonathan Uesato, Richard Ngo, Tom Everitt, Victoria Krakovna, et al. (2020). REALab: An Embedded Perspective on Tampering . arXiv: 2011.08820 . · arXiv
- Langlois, Eric and Tom Everitt (2021). 'How RL Agents Behave When Their Actions Are Modified'. In: AAAI . arXiv: 2102.07716 . · arXiv
- Lattimore, Tor and Marcus Hutter (2014). 'General time consistent discounting'. In: Theoretical Computer Science 519, pp. 140-154. arXiv: 1107.5528 . · arXiv
- Lauritzen, Steffen L. and Dennis Nilsson (2001). 'Representing and Solving Decision Problems with Limited Information'. In: Management Science 47.9, pp. 1235-1251.
- LaVictoire, Patrick, Benya Fallenstein, Eliezer S Yudkowsky, Mihaly Barasz, Paul Christiano, et al. (2014). 'Program Equilibrium in the Prisoner's Dilemma via L¨ ob's Theorem'. In: AAAI Workshop on Multiagent Interaction without Prior Coordination .
- Leike, Jan, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, et al. (2018). Scalable agent alignment via reward modeling: a research direction . arXiv: 1811.07871 . · arXiv
- Leike, Jan, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, et al. (2017). AI Safety Gridworlds . arXiv: 1711.09883 . · arXiv
- Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems . arXiv: 2005.01643 . · arXiv
- Milli, Smitha, Luca Belli, and Moritz Hardt (2020). 'From Optimizing Engagement to Measuring Value'. In: FAccT . arXiv: 2008.12623 . · arXiv
- Milli, Smitha, Dylan Hadfield-Menell, Anca Dragan, and Stuart J Russell (2017). 'Should robots be obedient?' In: IJCAI , pp. 4754-4760. isbn : 9780999241103. arXiv: 1705.09990 . · arXiv
- Ng, Andrew Y and Stuart J Russell (2000). 'Algorithms for inverse reinforcement learning'. In: Proceedings of the Seventeenth International Conference on Machine Learning , pp. 663-670.
- Olds, James and Peter Milner (1954). 'Positive Reinforcement Produced by Electrical Stimulation of Septal Area and other Regions of Rat Brain.' In: Journal of Comparative and Physiological Psychology 47.6, pp. 419-427.
- Omohundro, Stephen M (2008). 'The Basic AI Drives'. In: Artificial General Intelligence . Ed. by P. Wang, B. Goertzel, and S. Franklin. Vol. 171. IOS Press, pp. 483493.
- Orseau, Laurent and Stuart Armstrong (2016). 'Safely interruptible agents'. In: 32nd Conference on Uncertainty in Artificial Intelligence.
- Orseau, Laurent and Mark Ring (2011). 'Self-modification and mortality in artificial agents'. In: Artificial General Intelligence . Vol. 6830 LNAI, pp. 1-10.
- Pearl, Judea (2009). Causality: Models, Reasoning, and Inference . 2 edition. Cambridge University Press. isbn : 9780521895606.
- Petersen, Steve (2021). 'Machines Learning Values'. In: Ethics of Artificial Intelligence . Oxford University Press.
- Reddy, Siddharth, Anca D. Dragan, Sergey Levine, Shane Legg, and Jan Leike (2020). 'Learning human objectives by evaluating hypothetical behavior'. In: ICML . arXiv: 1912.05652 . · arXiv
- Ring, Mark and Laurent Orseau (2011). 'Delusion, Survival, and Intelligent Agents'. In: Artificial General Intelligence . Springer Berlin Heidelberg, pp. 1-11.
- Schmidhuber, J¨ urgen (2007). 'G¨ odel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements'. In: Artificial General Intelligence . Springer. arXiv: 0309048 [cs] .
- Shah, Rohin, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca D. Dragan (2019). 'Preferences implicit in the state of the world'. In: 7th International Conference on Learning Representations, ICLR . arXiv: 1902.04198 . · arXiv
- Shpitser, Ilya and Judea Pearl (2007). 'What counterfactuals can be tested'. In: Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, UAI 2007 , pp. 352-359. isbn : 0974903930.
- (2008). 'Complete identification methods for the causal hierarchy'. In: Journal of Machine Learning Research 9, pp. 1941-1979.
- Soares, Nate, Benya Fallenstein, Eliezer S Yudkowsky, and Stuart Armstrong (2015). 'Corrigibility'. In: AAAI Workshop on AI and Ethics , pp. 74-82.
- Sutton, Richard S and Andrew G Barto (2018). Reinforcement Learning: An Introduction . 2nd. MIT Press. isbn : 9780262039246.
- Turner, Alexander Matt, Dylan Hadfield-Menell, and Prasad Tadepalli (2020). 'Conservative agency via Attainable Utility Preservation'. In: AI, Ethics, and Society . arXiv: 1902.09725 . · arXiv
- Uesato, Jonathan, Ramana Kumar, Victoria Krakovna, Tom Everitt, Richard Ngo, et al. (2020). Avoiding Tampering Incentives in Deep RL via Decoupled Approval . arXiv: 2011.08827 . · arXiv
Cited by
Sorted by citations
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68 · see citing works
The Challenge of Value Alignment: From Fairer Algorithms to AI Safety
Gabriel, I., & Ghazavi, V. (2022) · Oxford Handbook of Digital Ethics · doi:10.1093/oxfordhb/9780198857815.013.18 · Cited by 52 · see citing works
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
Kenton, Z., Kumar, R., Farquhar, S., Richens, J., MacDermott, M., & Everitt, T. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103963 · Cited by 17 · see citing works
Self-citationHidden Incentives for Auto-Induced Distributional Shift
Krueger, D., Maharaj, T., & Leike, J. (2020) · arXiv · doi:10.48550/arxiv.2009.09153 · Cited by 14 · see citing works
- Authors
- Everitt, T., Hutter, M., Kumar, R., & Krakovna, V.
- Affiliations
- Australian National University (AU)
- Google (United States) (US)
- Corresponding author
- Everitt, Tom
- Venue
- Synthese
- Published
- 2021-05-19
- Volume
- 198
- Issue
- S27
- Pages
- 6435-6467
- ISSN
- 0039-7857 (ISSN-L)
- 1573-0964
- OpenAlex
- W2968771440
- Cited by
- 80
- References
- 7
- Citations by year
- 2026: 22 · 2025: 10 · 2024: 3 · 2023: 20 · 2022: 6 · 2021: 12 · 2020: 5 · 2019: 1 · 2018: 1
- FWCI
- 3.45
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Synthese (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- instrumental goals, safe reinforcement learning, artificial general intelligence, reward hacking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
