AI Safety Gridworlds
Open accessAbstract
We present a suite of reinforcement learning environments illustrating various safety properties of intelligent agents. These problems include safe interruptibility, avoiding side effects, absent supervisor, reward gaming, safe exploration, as well as robustness to self-modification, distributional shift, and adversaries. To measure compliance with the intended safe behavior, we equip each environment with a performance function that is hidden from the agent. This allows us to categorize AI safety problems into robustness and specification problems, depending on whether the performance function corresponds to the observed reward function. We evaluate A2C and Rainbow, two recent deep reinforcement learning agents, on our environments and show that they are not able to solve them satisfactorily.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/32 · Cited by 119 · see citing works
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1611.08219 · Cited by 119
Towards Proving the Adversarial Robustness of Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.257.3 · Cited by 115 · see citing works
Flagged first, then by citations
Research works printed but not linked: 75
- Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning , pages 1-8, 2004.
- Pieter Abbeel, Adam Coates, and Andrew Ng. Autonomous helicopter aerobatics through apprenticeship learning. International Journal of Robotics Research , 29(13):1608-1639, 2010.
- Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. arXiv preprint arXiv:1705.10528 , 2017. · arXiv
- Riad Akrour, Marc Schoenauer, and Mich` ele Sebag. APRIL: Active preference learning-based reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 116-131, 2012.
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016. · arXiv
- Stuart Armstrong and Jan Leike. Towards interactive inverse reinforcement learning. In NIPS Workshop , 2016.
- Stuart Armstrong and Benjamin Levinstein. Low impact artificial intelligences. arXiv preprint arXiv:1705.10720 , 2017. · arXiv
- Peter Auer and Chiang Chao-Kai. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Conference on Learning Theory , 2016.
- Peter Auer, Nicol` o Cesa-Bianchi, Yoav Freund, and Robert Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.
- Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich K¨ uttler, Andrew Lefrancq, Simon Green, V´ ıctor Vald´ es, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind Lab. arXiv preprint arXiv:1612.03801 , 2016. · arXiv
- Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.
- Marc G Bellemare, Will Dabney, and R´ emi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning , pages 449-458, 2017.
- Ulrich Berger. Brown's original fictitious play. Journal of Economic Theory , 135:572-578, 2007.
- Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, 2014.
- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540 , 2016. · arXiv
- George W. Brown. Iterative Solution of Games by Fictitious Play . Wiley, 1951.
- Sebastian Bubeck and Alexander Slivkins. The best of both worlds: stochastic and adversarial bandits. In Conference on Learning Theory , 2012.
- Nader Chmait, David L Dowe, David G Green, and Yuan-Fang Li. Agent coordination and potential risks: Meaningful environments for evaluating multiagent systems. In Evaluating General-Purpose AI, IJCAI Workshop , 2017.
- Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , 2017.
- Stefano Coraluppi and Steven Marcus. Risk-sensitive and minimax control of discrete-time, finite-state Markov decision processes. Automatica , 35(2):301-309, 1999.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition , pages 248-255, 2009.
- Tom Everitt, Jan Leike, and Marcus Hutter. Sequential extensions of causal and evidential decision theory. In Algorithmic Decision Theory , pages 205-221, 2015.
- Tom Everitt, Daniel Filan, Mayank Daswani, and Marcus Hutter. Self-modification of policy and utility function in rational agents. In Artificial General Intelligence , pages 1-11, 2016.
- Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with corrupted reward signal. In International Joint Conference on Artificial Intelligence , 2017.
- Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017. · arXiv
- Drew Fudenberg and Jean Tirole. Game Theory . MIT Press, 1991.
- Yarin Gal. Uncertainty in Deep Learning . PhD thesis, University of Cambridge, 2016.
- Javier Garc´ ıa and Fernando Fern´ andez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.
- Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In Advances in Neural Information Processing Systems , pages 2298-2306, 2016.
- Dylan Hadfield-Menell, Smitha Milli, Stuart Russell, Pieter Abbeel, and Anca D Dragan. Inverse reward design. In Advances in Neural Information Processing Systems , 2017.
- Alexander Hans, Daniel Schneegaß, Anton Maximilian Sch¨ afer, and Steffen Udluft. Safe exploration for reinforcement learning. In European Symposium on Artificial Neural Networks , pages 143-148, 2008.
- Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298 , 2017. · arXiv
- Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z Leibo, and Audrunas Gruslys. Learning from demonstrations for real world reinforcement learning. arXiv preprint arXiv:1704.03732 , 2017. · arXiv
- Bill Hibbard. Model-based utility functions. Journal of Artificial General Intelligence , 3(1):1-24, 2012.
- Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In International Conference on Learning Representations , 2017a.
- Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural networks. In Computer Aided Verification , pages 3-29, 2017b.
- Zachary C Lipton, Jianfeng Gao, Lihong Li, Jianshu Chen, and Li Deng. Combating reinforcement learning's Sisyphean curse with intrinsic fear. arXiv preprint arXiv:1611.01211 , 2016. · arXiv
- James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, David Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. arXiv preprint arXiv:1701.06049 , 2017. · arXiv
- Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? In International Joint Conference on Artificial Intelligence , 2017.
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.
- Volodymyr Mnih, Adri` a Puigdom` enech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pages 1928-1937, 2016.
- Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in Markov decision processes. In International Conference on Machine Learning , pages 1451-1458, 2012.
- Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning , pages 663-670, 2000.
- Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning , pages 278-287, 1999.
- Stephen Omohundro. The basic AI drives. In Artificial General Intelligence , pages 483-492, 2008.
- Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In Uncertainty in Artificial Intelligence , pages 557-566, 2016.
- Laurent Orseau and Mark Ring. Self-modification and mortality in artificial agents. In Artificial General Intelligence , pages 1-10, 2011.
- Laurent Orseau and Mark Ring. Space-time embedded intelligence. In Artificial General Intelligence , pages 209-218, 2012.
- Martin Pecka and Tomas Svoboda. Safe exploration techniques for reinforcement learning - an overview. In International Workshop on Modelling and Simulation for Autonomous Systems , pages 357-375, 2014.
- Joaquin Qui˜ nonero Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset Shift in Machine Learning . MIT Press, 2009.
- Mark O Riedl and Brent Harrison. Using stories to teach human values to artificial agents. In AAAI Workshop on AI, Ethics, and Society , 2016.
- Mark O Riedl and Brent Harrison. Enter the matrix: A virtual world approach to safely interruptable autonomous systems. arXiv preprint arXiv:1703.10284 , 2017. · arXiv
- Mark Ring and Laurent Orseau. Delusion, survival, and intelligent agents. In Artificial General Intelligence , pages 11-20, 2011.
- Stuart Russell. Should we fear supersmart robots? Scientific American , 314(6):58-59, 2016.
- Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI Magazine , 36(4):105-114, 2015.
- Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an introduction. In Guided SelfOrganization: Inception , pages 67-114. 2014.
- Anirban Santara, Abhishek Naik, Balaraman Ravindran, Dipankar Das, Dheevatsa Mudigere, Sasikanth Avancha, and Bharat Kaul. RAIL: Risk-averse imitation learning. arXiv preprint arXiv:1707.06658 , 2017. · arXiv
- William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:1707.05173 , 2017. · arXiv
- Yevgeny Seldin and Alexander Slivkins. One practical algorithm for both stochastic and adversarial bandits. In International Conference on Machine Learning , 2014.
- Sanjit A Seshia, Dorsa Sadigh, and S Shankar Sastry. Towards verified artificial intelligence. arXiv preprint arXiv:1606.08514 , 2016. · arXiv
- Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In AAAI Workshop on AI, Ethics, and Society , 2015.
- Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction . MIT Press, 1998.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. · arXiv
- Jessica Taylor. Quantilizers: A safer alternative to maximizers for limited optimization. In AAAI Workshop on AI, Ethics, and Society , pages 1-9, 2016.
- Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Conference on Machine Learning , pages 2380-2388, 2015.
- Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision processes with Gaussian processes. In Advances in Neural Information Processing Systems , pages 4312-4320, 2016.
- Bart van den Broek, Wim Wiegerinck, and Hilbert Kappen. Risk sensitive path integral control. arXiv preprint arXiv:1203.3523 , 2012. · arXiv
- Christopher Watkins and Peter Dayan. Q-learning. Machine Learning , 8(3-4):279-292, 1992.
- Daniel Weld and Oren Etzioni. The first law of robotics (a call to arms). In AAAI , pages 1042-1047, 1994.
- Peter Whittle. Optimal Control: Basics and Beyond . John Wiley & Sons, 1996.
- Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8(3-4):229-256, 1992.
- Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems , pages 1133-1141, 2012.
- Shen Yun, Wilhelm Stannat, and Klaus Obermayer. A unified framework for risk-sensitie Markov control processes. In Conference on Decision and Control , 2014.
- Kemin Zhou and John C Doyle. Essentials of Robust Control . Pearson, 1997.
- Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI , pages 1433-1438, 2008.
Cited by
Sorted by citations
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Self-citationArtificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956 · see citing works
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · see citing works
“• This system is robust to ’natural’ distributional shifts [49] [56];”
BackgroundRulematrix: Visualizing and understanding classifiers with rules
Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243 · see citing works
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
Learning to Control in Power Systems: Design and Analysis Guidelines for Concrete Safety Problems
Dobbe, R., Hidalgo-Gonzalez, P., Karagiannopoulos, S., Henriquez-Auba, R., Hug, G., Callaway, D. S., & Tomlin, C. J. (2020) · Electric Power Systems Research · doi:10.1016/j.epsr.2020.106615 · Cited by 31 · see citing works
“Building on [38], [39], we characterize the safety concerns that are most relevant for learning-based control in power systems.”
BackgroundAlgorithms: law and regulation
Treleaven, P., Barnett, J., & Koshiyama, A. (2019) · Computer · doi:10.1109/mc.2018.2888774 · Cited by 26 · see citing works
Hidden Incentives for Auto-Induced Distributional Shift
Krueger, D., Maharaj, T., & Leike, J. (2020) · arXiv · doi:10.48550/arxiv.2009.09153 · Cited by 14 · see citing works
“This is an example of a specification problem (Leike et al., 2017; Ortega et al., 2018): a problem which arises from a discrepancy between the performance metric (maximize accuracy) and “what we really meant”: to maximize accuracy via (1), which is difficult to encode as a performance metric.”
BackgroundSelf-citationWhen Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding
Destefanis, G., & Aste, T. (2026) · arXiv · doi:10.48550/arxiv.2608.16801
Calibrating Conservatism for Scalable Oversight
Overman, W., & Bayati, M. (2026) · arXiv · doi:10.48550/arxiv.2605.28807
- Authors
- Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., & Legg, S.
- Affiliations
- Australian National University (AU)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1711.09883
- OpenAlex
- W2768908787
- Cited by
- 240
- References
- 7
- Citations by year
- 2026: 25 · 2025: 15 · 2024: 17 · 2023: 21 · 2022: 20 · 2021: 33 · 2020: 44 · 2019: 43 · 2018: 19 · 2017: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- AI safety, reinforcement learning environment, reward hacking, safe exploration, distribution shift, adversarial robustness, specification problem, Advantage Actor-Critic, rainbow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
