The Off-Switch Game
Open accessAbstract
It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off. We analyze a simple game between a human H and a robot R, where H can press R's off switch but R can disable the off switch. A traditional agent takes its reward function for granted: we show that such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility. (R also has no incentive to switch itself off in this setting.) We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In a theoretical model, agents disable their off-switches.
“In a theoretical model, Hadfield-Menell et al. [2016] showed agents disabling their off-switches.”
From The alignment problem from a deep learning perspective (2022) · cites this paper as Hadfield-Menell et al., 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Artificial Intelligence: A Modern Approach
Ligęza, A. (1995) · Neurocomputing · doi:10.1016/0925-2312(95)90020-9 · Cited by 1,857 · see citing works
On the folly of rewarding A, while hoping for B
Kerr, S. (1975) · Academy of Management Journal · doi:10.2307/255378 · Cited by 1,585 · see citing works
Distortion and Risk in Optimal Incentive Contracts
Baker, G. P. (2002) · The Journal of Human Resources · doi:10.2307/3069615 · Cited by 493 · see citing works
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Self-citationCognition and Incomplete Contracts
Jean Tirole, Z. (2009) · American Economic Review · doi:10.1257/aer.99.1.265 · Cited by 269 · see citing works
Omohundro, S. M. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-3 · Cited by 148 · see citing works
Gibbons, R. S. (1998) · National Bureau of Economic Research · doi:10.3386/w6695 · Cited by 99 · see citing works
Should We Fear Supersmart Robots?
Russell, S. (2016) · Scientific American · doi:10.1038/scientificamerican0616-58 · Cited by 60 · see citing works
Self-citationDewey, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22887-2_35 · Cited by 52 · see citing works
A Decision-Theoretic Model of Assistance
Fern, A., Natarajan, S., Judah, K., & Tadepalli, P. V. (2014) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4213 · Cited by 38 · see citing works
Flagged first, then by citations
1 of the 11 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Ar- tificial Intelligence, 2015.
- Tirole, 2009] Jean Tirole. Cognition and incomplete con- tracts. The American Economic Review, 99(1):265-294, 2009.
Cited by
Sorted by citations
Russell, S. (2020) · doi:10.1093/oso/9780190905033.003.0012 · Cited by 289 · see citing works
Self-citationHybrid collective intelligence in a human–AI society
Peeters, M. M. M., van Diggelen, J., van den Bosch, K., Bronkhorst, A., Neerincx, M. A., Schraagen, J. M., & Raaijmakers, S. (2021) · AI & SOCIETY · doi:10.1007/s00146-020-01005-y · Cited by 229 · see citing works
Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., & Legg, S. (2017) · arXiv · doi:10.48550/arxiv.1711.09883 · Cited by 240 · see citing works
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., & Dragan, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02827 · Cited by 125 · see citing works
Self-citationCyber-security and reinforcement learning — A brief survey
Adawadkar, A. M. K., & Kulkarni, N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105116 · Cited by 123 · see citing works
Scalable agent alignment via reward modeling: a research direction
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018) · arXiv · doi:10.48550/arxiv.1811.07871 · Cited by 121 · see citing works
Perspectives on Digital Humanism
Werthner, H., Prem, E., Lee, E. A., & Ghezzi, C. (2021) · Springer eBooks · doi:10.1007/978-3-030-86144-5 · Cited by 117 · see citing works
Artificial Intelligence and the Problem of Control
Russell, S. (2021) · Springer eBooks · doi:10.1007/978-3-030-86144-5_3 · Cited by 100 · see citing works
Self-citationCollecting the Public Perception of AI and Robot Rights
Lima, G., Kim, C., Ryu, S., Jeon, C., & Cha, M. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415206 · Cited by 95 · see citing works
Unsolved Problems in ML Safety
Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2109.13916 · Cited by 95 · see citing works
The “big red button” is too late: an alternative model for the ethical evaluation of AI systems
Arnold, T., & Scheutz, M. (2018) · Ethics and Information Technology · doi:10.1007/s10676-018-9447-7 · Cited by 94 · see citing works
Special Issue “On Defining Artificial Intelligence”—Commentaries and Author’s Response
Monett, D., Lewis, C. W. P., Þórisson, K. R., Bach, J., Baldassarre, G., Granato, G., Berkeley, I. S. N., Chollet, F., Crosby, M., Shevlin, H., et al. (2020) · Journal of Artificial General Intelligence · doi:10.2478/jagi-2020-0003 · Cited by 98 · see citing works
Chatila, R., Dignum, V., Fisher, M., Giannotti, F., Morik, K. J., Russell, S., & Yeung, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69128-8_2 · Cited by 82 · see citing works
Self-citationFoundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf Restraining Specifications
Giacomo, G. D., Iocchi, L., Favorito, M., & Patrizi, F. (2019) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v29i1.3549 · Cited by 69 · see citing works
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68 · see citing works
Langley, C., Cirstea, B. I., Fabio Cuzzolin, A., & Sahakian, B. J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.778852 · Cited by 64 · see citing works
The Challenge of Value Alignment: From Fairer Algorithms to AI Safety
Gabriel, I., & Ghazavi, V. (2022) · Oxford Handbook of Digital Ethics · doi:10.1093/oxfordhb/9780198857815.013.18 · Cited by 52 · see citing works
Milli, S., Hadfield-Menell, D., Dragan, A. D., & Russell, S. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/662 · Cited by 46 · see citing works
Self-citationKenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., & Irving, G. (2021) · arXiv · doi:10.48550/arxiv.2103.14659 · Cited by 45 · see citing works
Social impact and governance of AI and neurotechnologies
Doya, K., Ema, A., Kitano, H., Sakagami, M., & Russell, S. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.05.012 · Cited by 45 · see citing works
Self-citationAI Alignment: A Contemporary Survey
Ji, J.-M., Qiu, T., Chen, B.-Y., Zhou, J.-Y., Zhang, B.-R., Hong, D., Lou, H., Wang, K.-L., Duan, Y., He, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3770749 · Cited by 57 · see citing works
Conservative Agency via Attainable Utility Preservation
Turner, A. M., Hadfield-Menell, D., & Tadepalli, P. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375851 · Cited by 36 · see citing works
Self-citationBeyond Preferences in AI Alignment
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 85 · see citing works
Learning to Control in Power Systems: Design and Analysis Guidelines for Concrete Safety Problems
Dobbe, R., Hidalgo-Gonzalez, P., Karagiannopoulos, S., Henriquez-Auba, R., Hug, G., Callaway, D. S., & Tomlin, C. J. (2020) · Electric Power Systems Research · doi:10.1016/j.epsr.2020.106615 · Cited by 31 · see citing works
Advanced Artificial Agents Intervene in the Provision of Reward
Cohen, M. K., Hutter, M., & Osborne, M. A. (2022) · AI Magazine · doi:10.1002/aaai.12064 · Cited by 30 · see citing works
Robot Planning with Mathematical Models of Human State and ActionPreprint
Dragan, A. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04226 · Cited by 29 · see citing works
Self-citationTowards an effective transnational regulation of AI
Gervais, D. J. (2021) · AI & Society · doi:10.1007/s00146-021-01310-0 · Cited by 28 · see citing works
Incomplete Contracting and AI Alignment
Hadfield-Menell, D., & Hadfield, G. K. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314250 · Cited by 26 · see citing works
Self-citationFeature Expansive Reward Learning
Bobu, A., Wiggert, M., Tomlin, C. J., & Dragan, A. D. (2021) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3434073.3444667 · Cited by 27 · see citing works
Self-citationImpossibility Results in AI: A Survey
Brčić, M., & Yampolskiy, R. V. (2023) · ACM Computing Surveys · doi:10.1145/3603371 · Cited by 25 · see citing works
The value alignment problem: a geometric approach
Peterson, M. (2018) · Ethics and Information Technology · doi:10.1007/s10676-018-9486-0 · Cited by 20 · see citing works
Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems
Subbarao Kambhampati, B., Sreedharan, S., Verma, M., Zha, Y., & Guan, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21488 · Cited by 20 · see citing works
Interaction Considerations in Learning from Humans
Koppol, P., Admoni, H., & Simmons, R. G. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/40 · Cited by 18 · see citing works
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action SettingsPreprint
Everitt, T., Ortega, P. A., Barnes, E. A., & Legg, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09980 · Cited by 27 · see citing works
Optimal Policies Tend To Seek Power
Turner, A. M., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2019) · arXiv · doi:10.48550/arxiv.1912.01683 · Cited by 18 · see citing works
On conflicts between ethical and logical principles in artificial intelligence
D’Acquisto, G. (2020) · AI & Society · doi:10.1007/s00146-019-00927-6 · Cited by 17 · see citing works
Butlin, P. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462570 · Cited by 16 · see citing works
Theories of Parenting and Their Application to Artificial Intelligence
Croeser, S., & Eckersley, P. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314231 · Cited by 16 · see citing works
Axes for Sociotechnical Inquiry in AI Research
Dean, S., Gilbert, T. K., Lambert, N., & Zick, T. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3074097 · Cited by 16 · see citing works
The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types
Ghosal, G. R., Zurek, M., Brown, D. S., & Dragan, A. D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25740 · Cited by 15 · see citing works
Self-citationReinforcement Learning Under Moral UncertaintyPreprint
Ecoffet, A., & Lehman, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04734 · Cited by 18 · see citing works
Zahedi, Z., Verma, M., Sreedharan, S., & Subbarao Kambhampati, B. (2023) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3568162.3578628 · Cited by 14 · see citing works
The Jiminy Advisor: Moral Agreements among Stakeholders Based on Norms and Argumentation
Liao, B., Pardo, P., Slavkovik, M., & van der Torre, L. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14368 · Cited by 13 · see citing works
Metaethical perspectives on ‘benchmarking’ AI ethics
LaCroix, T., & Luccioni, A. S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00703-x · Cited by 16 · see citing works
Incentives for responsiveness, instrumental control and impact
Carey, R., Langlois, E., van Merwijk, C., Legg, S., & Everitt, T. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2025.104408 · Cited by 13 · see citing works
Designing Preferences, Beliefs, and Identities for Artificial Intelligence
Conitzer, V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019755 · Cited by 13 · see citing works
Is there a tension between AI safety and AI welfare?
Long, R., Sebo, J., & Sims, T. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02302-2 · Cited by 12 · see citing works
Alon, N., Schulz, L., Rosenschein, J. S., & Dayan, P. (2023) · Open Mind · doi:10.1162/opmi_a_00097 · Cited by 12 · see citing works
Socialni odnosi, animizem, antropomorfizem in interakcija z UI
Babić, N. Č. (2022) · doi:10.18690/um.ff.11.2022.12 · Cited by 10 · see citing works
Phelps, S., & Russell, Y. I. (2025) · Journal of Physics Complexity · doi:10.1088/2632-072x/ada711 · Cited by 35 · see citing works
The shutdown problem: an AI engineering puzzle for decision theorists
Thornley, E. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02153-3 · Cited by 34 · see citing works
Unifying Principles and Metrics for Safe and Assistive AI
Srivastava, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17769 · Cited by 9 · see citing works
A Game-Theoretic Analysis of the Off-Switch Game
Wängberg, T., Böörs, M., Catt, E., Everitt, T., & Hütter, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-63703-7_16 · Cited by 9 · see citing works
Self-Regulating Artificial General Intelligence
Gans, J. S. (2018) · National Bureau of Economic Research · doi:10.3386/w24352 · Cited by 9 · see citing works
Zahedi, Z., Sengupta, S., & Subbarao Kambhampati, B. (2024) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2024.3488559 · Cited by 8 · see citing works
Chow, E., Lu, T. T., Payumo, K., de Fourtou, G. B., Sadler, E., Janvisloo, N. E., Carrillo, J., Li, B., Hubler, B., Littlejohn, O., et al. (2024) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero58975.2024.10521103 · Cited by 8 · see citing works
Towards AGI Agent Safety by Iteratively Improving the Utility Function
Holtman, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52152-3_21 · Cited by 8 · see citing works
Curiosity-Driven Reinforced Learning of Undesired Actions in Autonomous Intelligent Agents
Rosser, C. C., & Abed, K. H. (2021) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami50585.2021.9378666 · Cited by 8 · see citing works
Optimal Farsighted Agents Tend to Seek Power.Preprint
Turner, A. (2019) · arXiv (Cornell University) · Cited by 8 · see citing works
On Controllability of AIPreprint
Yampolskiy, R. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.04071 · Cited by 8 · see citing works
Why do experts disagree on existential risk? A survey of AI experts
Field, S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00762-0 · Cited by 7 · see citing works
A Generalized Acquisition Function for Preference-based Reward Learning
Ellis, E., Ghosal, G. R., Russell, S., Dragan, A. D., & Bıyık, E. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611472 · Cited by 7 · see citing works
Self-citationTrust-Aware Planning: Modeling Trust Evolution in Longitudinal Human-Robot Interaction.Preprint
Zahedi, Z., Verma, M., Sreedharan, S., & Subbarao Kambhampati, B. (2021) · arXiv (Cornell University) · Cited by 7 · see citing works
Democratizing traffic control in smart cities
Korecki, M., Dailisan, D. N., Yang, J. C., & Helbing, D. (2024) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2024.104511 · Cited by 6 · see citing works
Towards Empathetic PlanningPreprint
Shvo, M., & McIlraith, S. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06436 · Cited by 6 · see citing works
Artificial superintelligence alignment in healthcare
Ueda, D., Walston, S. L., Kurokawa, R., Saida, T., Honda, M., Iima, M., Watabe, T., Yanagawa, M., Nishioka, K., Sofue, K., et al. (2025) · Japanese Journal of Radiology · doi:10.1007/s11604-025-01907-1 · Cited by 5 · see citing works
Human-robot interaction through adjustable social autonomy
Cantucci, F., Falcone, R., & Castelfranchi, C. (2022) · Intelligenza Artificiale · doi:10.3233/ia-210124 · Cited by 5 · see citing works
Goldstein, S., & Robinson, P. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02099-6 · Cited by 4 · see citing works
Action Guidance and AI Alignment
Robinson, P. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604714 · Cited by 3 · see citing works
Challenges for Using Impact Regularizers to Avoid Negative Side Effects
Lindner, D., Matoba, K., & Meulemans, A. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000529308 · Cited by 3 · see citing works
Vloga umetne inteligence v izobraževanju in za izobraževanje
Pesek, I., & Krašna, M. (2022) · doi:10.18690/um.ff.11.2022.13 · Cited by 3 · see citing works
Varricchione, G., Alechina, N., Dastani, M. M., Giacomo, G. D., Logan, B. S., & Perelli, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30163 · Cited by 3 · see citing works
To Monitor or to Trust: Observing Robot's Behavior based on a Game-Theoretic Model of Trust.Preprint
Sengupta, S., Zahedi, Z., & Subbarao Kambhampati, B. (2019) · arXiv (Cornell University) · Cited by 3 · see citing works
Artificial Intelligence: Approaches to Safety
D’Alessandro, W., & Kirk‐Giannini, C. D. (2025) · Philosophy Compass · doi:10.1111/phc3.70039 · Cited by 2 · see citing works
AI alignment is all your need for future drug discovery
Li, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1668794 · Cited by 2 · see citing works
Automation, Alignment, and the Cooperative Interface
Jonker, J. D. (2023) · The Journal of Ethics · doi:10.1007/s10892-023-09449-2 · Cited by 2 · see citing works
Neth, S. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02296-x · Cited by 2 · see citing works
Agent Incentives: A Causal Perspective
Everitt, T., Carey, R., Langlois, E. D., Ortega, P. A., & Legg, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17368 · Cited by 2 · see citing works
Anić, N., & Anić, P. (2021) · National security and the future · doi:10.37458/nstf.21.3.4 · Cited by 2 · see citing works
Virtuously Safe Reinforcement LearningPreprint
Aslund, H., Mhamdi, E. M. E., Guerraoui, R., & Maurer, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11447 · Cited by 2 · see citing works
Self-Regulating Artificial General IntelligencePreprint
Gans, J. S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3124512 · Cited by 2 · see citing works
Ismail, M. S. (2024) · Mathematical Social Sciences · doi:10.1016/j.mathsocsci.2024.03.004 · Cited by 1 · see citing works
Automated Synthesis of Verified Neural Network Controllers from Linear Temporal Logic Specifications
O’Quinn, A., & Taylor, M. (2025) · Proceedings of the 2025 Workshop on Re-design Industrial Control Systems with Security · doi:10.1145/3733823.3764517 · Cited by 1 · see citing works
HADA: Human-AI Agent Decision Alignment Architecture
Pitkäranta, T., & Pitkäranta, L. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_5 · Cited by 1 · see citing works
The self-preservation test for artificial sentience
Mullally, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-00983-x · Cited by 1 · see citing works
Sodobne perspektive družbe: Umetna inteligenca na stičišču znanosti
Bregant, J., Aberšek, B., & Borstner, B. (2022) · doi:10.18690/um.ff.11.2022 · Cited by 1 · see citing works
Unveiling the Transformative Power
Vetrivel, S. C., Vidhyapriya, P., & Arun, V. P. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-5533-6.ch001 · Cited by 1 · see citing works
Ethically Aligned Design in Autonomous and Intelligent Systems: An Overview
Burnside, A., & Bodde, E. R. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098316 · Cited by 1 · see citing works
Enter the Matrix: Safely Interruptible Autonomous Systems via VirtualizationPreprint
Riedl, M. O., & Harrison, B. E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10284 · Cited by 1 · see citing works
A Roadmap for Robust End-to-End AlignmentPreprint
Hoang, L. N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01036 · Cited by 1 · see citing works
Democratizing Traffic Control in Smart CitiesPreprint
Korecki, M., Dailisan, D. N., Yang, J. C., & Helbing, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4598461 · Cited by 1 · see citing works
Understanding the Concept of Knowledge in Healthcare Services: A Grounded Approach
El-Ghariani, K. A. B. (2019) · Sheffield Hallam University · doi:10.7190/shu-thesis-00209
Pavlić, M. (2020) · Zbornik radova Fakulteta tehničkih nauka u Novom Sadu · doi:10.24867/11be02pavlic
Risk-Averse Biased Human Policies with a Robot Assistant in Multi-Armed Bandit Settings
Koller, M., Patten, T., & Vincze, M. (2021) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3453892.3461628
Umetna inteligenca in eksistenčno tveganje
Lipuš, A. (2022) · doi:10.18690/um.ff.11.2022.10
Borstner, B., & Šetar, N. (2022) · doi:10.18690/um.ff.11.2022.5
- Authors
- Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S.
- Affiliations
- University of California, Berkeley (US)
- International Computer Science Institute (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- Pages
- 220-227
- arXiv
- 1611.08219
- OpenAlex
- W2558801266
- Cited by
- 119
- References
- 11
- Citations by year
- 2026: 16 · 2025: 13 · 2024: 10 · 2023: 10 · 2022: 18 · 2021: 17 · 2020: 11 · 2019: 11 · 2018: 7 · 2017: 6
- FWCI
- 23.89
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- National Conference on Artificial Intelligence
- Funders
- National Science Foundation (1106400, 1652083)
- Open Philanthropy Project
- Future of Life Institute
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- AI safety, self-preservation, human-robot interaction, rational agents, expected utility maximization, reward function
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
