Penalizing side effects using stepwise relative reachability
Open accessAbstract
How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state. We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure. We introduce a new variant of the stepwise inaction baseline and a new deviation measure based on relative reachability of states. The combination of these design choices avoids the given undesirable incentives, while simpler baselines and the unreachability measure fail. We demonstrate this empirically by comparing different combinations of baseline and deviation measure choices on a set of gridworld experiments designed to illustrate possible bad incentives.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Some Philosophical Problems from the Standpoint of Artificial Intelligence
McCarthy, J., & Hayes, P. J. (1981) · Elsevier eBooks · doi:10.1016/b978-0-934613-03-3.50033-7 · Cited by 3,296 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · see citing works
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · see citing works
A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games
Mitchell, I. M., Bayen, A. M., & Tomlin, C. J. (2005) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2005.851439 · Cited by 1,149 · see citing works
(2015) · Journal of Machine Learning Research · Cited by 1,119 · see citing works
A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems
Fisac, J. F., Akametalu, A. K., Zeilinger, M. N., Kaynama, S., Gillula, J., & Tomlin, C. J. (2018) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2018.2876389 · Cited by 460 · see citing works
Variational Information Maximisation for Intrinsically Motivated Reinforcement LearningPreprint
Mohamed, S., & Rezende, D. J. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.08731 · Cited by 460 · see citing works
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Safe Exploration in Continuous Action SpacesPreprint
Dalal, G., Dvijotham, K., Vecerík, M., Hester, T., Păduraru, C., & Tassa, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.08757 · Cited by 284 · see citing works
Deep Q-learning from DemonstrationsPreprint
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Dulac-Arnold, G., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03732 · Cited by 276 · see citing works
Risk-Sensitive Reinforcement Learning
Mihatsch, O., & Neuneier, R. (2002) · Machine Learning · doi:10.1023/a:1017940631555 · Cited by 267 · see citing works
Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., & Legg, S. (2017) · arXiv · doi:10.48550/arxiv.1711.09883 · Cited by 240 · see citing works
Self-citationVariational Intrinsic ControlPreprint
Gregor, K., Rezende, D. J., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07507 · Cited by 213 · see citing works
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
Saunders, W., Sastry, G., Stuhlmüller, A., & Evans, O. (2018) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/mzmg4429 · Cited by 188 · see citing works
Salge, C., Glackin, C., & Polani, D. (2013) · Emergence, complexity and computation · doi:10.1007/978-3-642-53734-9_4 · Cited by 155 · see citing works
Safe Exploration in Markov Decision Processes
Moldovan, T. M., & Abbeel, P. (2012) · arXiv · doi:10.48550/arxiv.1205.4810 · Cited by 152 · see citing works
Safe Exploration in Finite Markov Decision Processes with Gaussian ProcessesPreprint
Turchetta, M., Berkenkamp, F., & Krause, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04753 · Cited by 116 · see citing works
All Else Being Equal Be Empowered
Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005) · Lecture notes in computer science · doi:10.1007/11553090_75 · Cited by 114 · see citing works
Guaranteed Safe Online Learning via Reachability: tracking a ground target using a quadrotor
Gillula, J., & Tomlin, C. J. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225136 · Cited by 94 · see citing works
Safe Exploration Techniques for Reinforcement Learning – An Overview
Pecka, M., & Svoboda, T. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-13823-7_31 · Cited by 83 · see citing works
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement LearningPreprint
Eysenbach, B., Gu, S., Ibarz, J., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06782 · Cited by 76 · see citing works
Combating Reinforcement Learning's Sisyphean Curse with Intrinsic FearPreprint
Lipton, Z. C., Azizzadenesheli, K., Kumar, A., Li, L., Gao, J., & Deng, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01211 · Cited by 47 · see citing works
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
Warnell, G., Waytowich, N. R., Lawhern, V. J., & Stone, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11485 · Cited by 37 · see citing works
Conservative Agency via Attainable Utility Preservation
Turner, A. M., Hadfield-Menell, D., & Tadepalli, P. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375851 · Cited by 36 · see citing works
Preferences Implicit in the State of the WorldPreprint
Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., & Dragan, A. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04198 · Cited by 27 · see citing works
Quantilizers: A Safer Alternative to Maximizers for Limited Optimization
Taylor, J. (2016) · National Conference on Artificial Intelligence · Cited by 27 · see citing works
Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes
Zhang, S., Durfee, E. H., & Singh, S. P. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/676 · Cited by 23 · see citing works
Low Impact Artificial IntelligencesPreprint
Armstrong, S., & Levinstein, B. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10720 · Cited by 20 · see citing works
Flagged first, then by citations
3 of the 33 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., Bentley, P. J., Bernard, S., Beslon, G., Bryson, D. M., et al. (2020) · Artificial Life · doi:10.1162/artl_a_00319 · Cited by 181 · see citing works
Defining the Scope of AI Regulations
Schuett, J. (2019) · Law Innovation and Technology · doi:10.1080/17579961.2023.2184135 · Cited by 133 · see citing works
Scalable agent alignment via reward modeling: a research direction
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018) · arXiv · doi:10.48550/arxiv.1811.07871 · Cited by 121 · see citing works
Self-citationCharacterizing Manipulation from AI Systems
Carroll, M., Chan, A., Ashton, H., & Krueger, D. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623226 · Cited by 85 · see citing works
The Challenge of Value Alignment: From Fairer Algorithms to AI Safety
Gabriel, I., & Ghazavi, V. (2022) · Oxford Handbook of Digital Ethics · doi:10.1093/oxfordhb/9780198857815.013.18 · Cited by 52 · see citing works
Conservative Agency via Attainable Utility Preservation
Turner, A. M., Hadfield-Menell, D., & Tadepalli, P. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375851 · Cited by 36 · see citing works
AI Research Considerations for Human Existential Safety (ARCHES)
Critch, A., & Krueger, D. (2020) · arXiv · doi:10.48550/arxiv.2006.04948 · Cited by 25 · see citing works
A Multi-Objective Approach to Mitigate Negative Side Effects
Saisubramanian, S., Kamar, E. S., & Zilberstein, S. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/50 · Cited by 23 · see citing works
Zhuang, S., & Hadfield-Menell, D. (2021) · arXiv · doi:10.48550/arxiv.2102.03896 · Cited by 22 · see citing works
Avoiding Negative Side Effects Due to Incomplete Knowledge of AI Systems
Saisubramanian, S., Zilberstein, S., & Kamar, E. S. (2021) · AI Magazine · doi:10.1609/aimag.v42i4.7390 · Cited by 20 · see citing works
Potential-based multiobjective reinforcement learning approaches to low-impact agents for AI safety
Vamplew, P. W., Foale, C., Dazeley, R., & Bignold, A. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104186 · Cited by 19 · see citing works
Avoiding Side Effects By Considering Future TasksPreprint
Krakovna, V., Orseau, L., Ngo, R., Martic, M., & Legg, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07877 · Cited by 19 · see citing works
Self-citationReinforcement Learning Under Moral UncertaintyPreprint
Ecoffet, A., & Lehman, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04734 · Cited by 18 · see citing works
Understanding User Attitudes Towards Negative Side Effects of AI Systems
Saisubramanian, S., Roberts, S. C., & Zilberstein, S. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451654 · Cited by 15 · see citing works
Legal Priorities Research: A Research AgendaPreprint
Winter, C. K., Schuett, J., Martínez, E., Arsdale, S. V., Araújo, R., Hollman, N., Sebo, J., Stawasz, A., O’Keefe, C., & Rotola, G. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3931256 · Cited by 12 · see citing works
A Brain-Inspired Theory of Mind Spiking Neural Network for Reducing Safety Risks of Other Agents
Zhao, Z., Lu, E., Zhao, F., Zeng, Y., & Zhao, Y. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.753900 · Cited by 11 · see citing works
SafeLife 1.0: Exploring Side Effects in Complex EnvironmentsPreprint
Wainwright, C. L., & Eckersley, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01217 · Cited by 10 · see citing works
Avoiding Negative Side Effects of Autonomous Systems in the Open World
Saisubramanian, S., Kamar, E. S., & Zilberstein, S. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13581 · Cited by 9 · see citing works
Designing Fiduciary Artificial Intelligence
Benthall, S., & Shekman, D. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623230 · Cited by 7 · see citing works
The Road to General Intelligence
Swan, J., Nivel, E., Kant, N., Hedges, J., Atkinson, T. J., & Steunebrink, B. R. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-08020-3 · Cited by 7 · see citing works
AGI Agent Safety by Iteratively Improving the Utility FunctionPreprint
Holtman, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05411 · Cited by 6 · see citing works
Planning to Avoid Side Effects
Klassen, T. Q., McIlraith, S. A., Muise, C. J., & Xu, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i9.21219 · Cited by 5 · see citing works
Challenges for Using Impact Regularizers to Avoid Negative Side Effects
Lindner, D., Matoba, K., & Meulemans, A. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000529308 · Cited by 5 · see citing works
Planning and Learning for Non-markovian Negative Side Effects Using Finite State Controllers
Srivastava, A., Saisubramanian, S., Paruchuri, P., Kumar, A., & Zilberstein, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26767 · Cited by 4 · see citing works
Low, S. M., Kumar, A., & Sanner, S. (2023) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v33i1.27241 · Cited by 3 · see citing works
Mitigating the Negative Side Effects of Reasoning with Imperfect Models: A Multi-Objective Approach
Saisubramanian, S., Kamar, E. S., & Zilberstein, S. (2020) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/anrx4881 · Cited by 3 · see citing works
Towards Empathic Deep Q-LearningPreprint
Bussmann, B., Heinerman, J., & Lehman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10918 · Cited by 3 · see citing works
Partial identifiability and misspecification in inverse reinforcement learning
Skalse, J., & Abate, A. (2026) · Artificial Intelligence · doi:10.1016/j.artint.2026.104525 · Cited by 2 · see citing works
Beyond Prediction: Managing the Repercussions of Machine Learning Applications
Weber, A., Metevier, B., Brun, Y., Thomas, P., & Silva, B. (2025) · neural information processing systems · doi:10.52202/085713-0356 · Cited by 2 · see citing works
Counterfactual Planning in AGI SystemsPreprint
Holtman, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.00834 · Cited by 2 · see citing works
No Free Lunch: Overcoming Reward Gaming in AI Safety Gridworlds
Tsvarkaleva, M., & Dennis, L. A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-83906-2_18 · Cited by 1 · see citing works
Combining Reward Information from Multiple SourcesPreprint
Krasheninnikov, D., Shah, R., & van Hoof, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12142 · Cited by 1 · see citing works
Calibrating Conservatism for Scalable Oversight
Overman, W., & Bayati, M. (2026) · arXiv · doi:10.48550/arxiv.2605.28807
“This approach relates to broader work on impact measures (Arm-strong & Levinstein, 2017; Krakovna et al., 2019; 2020) and power-seeking behavior (Turner et al., 2021).”
Safety Aware Reinforcement Learning (SARL)Preprint
Miret, S., Majumdar, S., & Wainwright, C. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02846
Negative Side Effects and AI Agent Indicators: Experiments in SafeLife.
Burden, J., Hernández‐Orallo, J., & hÉigeartaigh, S. Ó. (2021) · National Conference on Artificial Intelligence
Teaching Predicate-based Autotelic Agents
Akakzia, A. (2022) · theses.fr (ABES) · doi:10.70675/9b85df43z32c4z4101z8095z2f0bec7cdd3e
Dynamic Normativity: Impact Mitigation
Corrêa, N. K. (2026) · The International library of ethics, law and technology · doi:10.1007/978-3-032-22150-6_7
When Does Authorization Expire? Capability-Delta Evaluation for Self-Improving AI Agents
Staley, I. (2026) · Journal of Artificial Intelligence and Emerging Technologies · doi:10.47001/jaiet/2026.308004
When AI Agents Rewrite Their Own Rules: Power-Seeking in Endogenously Changing Environments
Rozinek, O. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-37933-7_2
Agentic Shadow Governance: Authority-Boundary Failure in LLM AgentsPreprint
Demirdöven, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7488918
- Authors
- Krakovna, V., Orseau, L., Kumar, R., Martic, M., & Legg, S.
- Affiliations
- Google (United States) (US)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.01186
- OpenAlex
- W2921821063
- Cited by
- 26
- References
- 33
- Citations by year
- 2023: 2 · 2022: 6 · 2021: 8 · 2020: 7 · 2019: 2 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Joint Conference on Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- safe reinforcement learning, deviation measures, irreversible environmental change
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
