Conservative Agency via Attainable Utility Preservation
Open accessAbstract
Reward functions are easy to misspecify; although designers can make corrections after observing mistakes, an agent pursuing a misspecified reward function can irreversibly change the state of its environment. If that change precludes optimization of the correctly specified reward function, then correction is futile. For example, a robotic factory assistant could break expensive equipment due to a reward misspecification; even if the designers immediately correct the reward function, the damage is done. To mitigate this risk, we introduce an approach that balances optimization of the primary reward function with preservation of the ability to optimize auxiliary reward functions. Surprisingly, even when the auxiliary reward functions are randomly generated and therefore uninformative about the correctly specified reward function, this approach induces conservative, effective behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AUP is a method for side-effect avoidance that penalizes actions which substantially change an agent's ability to optimize a diverse set of auxiliary objectives, thereby encouraging conservative, reversible behavior.
“Attainable Utility Preservation (AUP) (Turner et al., 2020b) is a method for side-effect avoidance that penalizes actions which substantially change an agent's ability to optimize a diverse set of auxiliary objectives, thereby encouraging conservative, reversible behavior.”
From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Turner et al., 2020b
The original AUP formulation requires computing auxiliary optimal Q-functions {Q_i}.
“The original AUP formulation has three key limitations: it requires computing auxiliary optimal Q-functions {Q i }, offers no principled mechanism for selecting λ to meet a desired safety target, and provides no formal guarantees on violation rates.”
From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Turner et al., 2020b
The original AUP formulation offers no principled mechanism for selecting λ to meet a desired safety target.
“The original AUP formulation has three key limitations: it requires computing auxiliary optimal Q-functions {Q i }, offers no principled mechanism for selecting λ to meet a desired safety target, and provides no formal guarantees on violation rates.”
From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Turner et al., 2020b
The original AUP formulation provides no formal guarantees on violation rates.
“The original AUP formulation has three key limitations: it requires computing auxiliary optimal Q-functions {Q i }, offers no principled mechanism for selecting λ to meet a desired safety target, and provides no formal guarantees on violation rates.”
From Calibrating Conservatism for Scalable Oversight (2026) · cites this paper as Turner et al., 2020b
The Formalization (p. 2) and Theoretical Results (p. 7) give convergence guarantees, while Experimental Design (p. 4) evaluates side effects empirically. I searched for violation-rate and safety guarantees, but found no passage that states whether the formulation provides a formal guarantee on violation rates. The absence of such a guarantee from the passages found does not itself verify the claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Self-citationHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/32 · Cited by 119 · see citing works
Self-citationHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1611.08219 · Cited by 119
Self-citation
Flagged first, then by citations
Research works printed but not linked: 23
- [Altman, 1999] Eitan Altman. Constrained Markov decision processes , volume 7. CRC Press, 1999.
- [Amodei et al. , 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in AI safety. arXiv:1606.06565 [cs] , June 2016. arXiv: 1606.06565. · arXiv
- [Armstrong and Levinstein, 2017] Stuart Armstrong and Benjamin Levinstein. Low impact artificial intelligences. arXiv:1705.10720 [cs] , May 2017. arXiv: 1705.10720. · arXiv
- [Berkenkamp et al. , 2017] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems , pages 908-918, 2017.
- [Chow et al. , 2018] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. In Advances in Neural Information Processing Systems , pages 8092-8101, 2018.
- [Christiano et al. , 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pages 4299-4307, 2017.
- [Everitt et al. , 2017] Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel. In Proceedings of the TwentySixth International Joint Conference on Artificial Intelligence, IJCAI-17 , pages 4705-4713, 2017.
- [Eysenbach et al. , 2018] Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations , 2018.
- [Garc´ ıa and Fern´ andez, 2015] Javier Garc´ ıa and Fernando Fern´ andez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.
- [Hadfield-Menell et al. , 2017b] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems , pages 6765-6774, 2017.
- [Krakovna et al. , 2018] Victoria Krakovna, Laurent Orseau, Miljan Martic, and Shane Legg. Measuring and avoiding side effects using relative reachability. arXiv:1806.01186 [cs, stat] , June 2018. arXiv: 1806.01186. · arXiv
- [Leike et al. , 2017] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv:1711.09883 [cs] , November 2017. arXiv: 1711.09883. · arXiv
- [Mohamed and Rezende, 2015] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems , pages 2125-2133, 2015.
- [Moldovan and Abbeel, 2012] Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in Markov decision processes. ICML , 2012.
- [Pecka and Svoboda, 2014] Martin Pecka and Tomas Svoboda. Safe exploration techniques for reinforcement learning-an overview. In International Workshop on Modelling and Simulation for Autonomous Systems , pages 357-375. Springer, 2014.
- [Regan and Boutilier, 2010] Kevin Regan and Craig Boutilier. Robust policy computation in reward-uncertain MDPs using nondominated policies. In AAAI , 2010.
- [Saunders et al. , 2018] William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error:
- Towards safe reinforcement learning via human intervention. In Proceedings of the 17th International Conference on Autonomous Agents and Multi-Agent Systems , pages 2067-2069, 2018.
- [Shah et al. , 2019] Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. The implicit preference information in an initial state. In International Conference on Learning Representations , 2019.
- [Silver et al. , 2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484, 2016.
- [Soares et al. , 2015] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. AAAI Workshops , 2015.
- [Watkins and Dayan, 1992] Christopher Watkins and Peter Dayan. Q-learning. Machine Learning , 8(3-4):279-292, 1992.
- [Zhang et al. , 2018] Shun Zhang, Edmund H Durfee, and Satinder P Singh. Minimax-regret querying on side effects for safe optimality in factored Markov decision processes. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI18 , pages 4867-4873, 2018.
Cited by
Sorted by citations
Calibrating Conservatism for Scalable Oversight
Overman, W., & Bayati, M. (2026) · arXiv · doi:10.48550/arxiv.2605.28807
“Attain-able Utility Preservation (AUP) (Turner et al., 2020a;b) generalizes this, penalizing actions that substantially change the agent’s ability to optimize auxiliary reward functions.”
Influential
- Authors
- Turner, A. M., Hadfield-Menell, D., & Tadepalli, P.
- Affiliations
- Oregon State University (US)
- University of California, Berkeley (US)
- Venue
- Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2020-02-07
- Pages
- 385-391
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W2916350282
- Cited by
- 36
- References
- 3
- Citations by year
- 2026: 8 · 2025: 1 · 2024: 3 · 2023: 3 · 2022: 4 · 2021: 8 · 2020: 5 · 2019: 3 · 2018: 1
- FWCI
- 1.91
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Long-Term Future Fund
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- irreversible environmental change, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
