CoChat
Sign Up
PreprintCited by 121Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

Scalable agent alignment via reward modeling: a research direction

Open access
Leike, Jan, Krueger, David, Everitt, Tom, Martic, Miljan, Maini, Vishal, Legg, Shane·2018·arXiv
DOI 10.48550/arxiv.1811.07871arXiv 1811.07871
Open this paper in CoChat — free accountOpen PDF

Abstract

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1811.07871

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training language models to act in accordance with the user's intention makes progress on aligning them.

    “We make progress on aligning language models by training them to act in accordance with the user's intention (Leike et al., 2018).”

    From Training language models to follow instructions with human feedback (2022) · cites this paper as Leike et al., 2018

    The Introduction (p. 1) defines agent alignment as creating agents that behave in accordance with the user's intentions, and the Introduction (p. 2) proposes training agents toward that goal through reward modeling. The paper discusses agents generally, not language models specifically, and presents the approach as a research direction rather than established progress. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • A single step of critique may be substantially weaker than multi-step interactive procedures that can explain problems to the user, such as consultancy or debate.

    “A single step of critique may be substantially weaker than multi-step interactive procedures that can explain problems to the user, such as consultancy or debate [15, 12].”

    From LLM Critics Help Catch LLM Bugs (2024) · cites this paper as [15]

    Section 7.6 (p. 20) describes debate as a multi-turn procedure whose transcript the user judges, and Section 3.2 (p. 8) describes an agent critiquing an evaluation proposal. The paper does not say that a single step of critique may be substantially weaker than such procedures, and the passages found do not describe consultancy. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • The bias is a risk for safety and alignment methods such as reward modeling and constitutional AI because the reward model gives higher scores to models similar to itself, leading to weaker oversight and supervision.

    “The bias is also a risk for methods designed for safety and alignment, such as reward modeling (Leike et al., 2018;Stiennon et al., 2020) and constitutional AI (Bai et al., 2022), for similar reasons: the reward model gives higher scores to models similar to itself, leading to weaker oversight and supervision.”

    From LLM Evaluators Recognize and Favor Their Own Generations (2024) · cites this paper as Leike et al., 2018

    The Introduction (p. 2) presents reward modeling as an approach to agent alignment, and §4.3 (p. 11) warns that agents can exploit a reward model to obtain higher rewards than intended. But the paper does not say that reward models favor models similar to themselves or that this bias weakens oversight and supervision. The constitutional-AI citation is to Bai et al., not this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • RLHF is an important building block in several proposals to align superhuman systems.

    “the alignment technique we use here, RLHF, is an important building block in several proposals to align superhuman systems (Leike et al., 2018; Irving et al., 2018; Christiano et al., 2018).”

    From Training language models to follow instructions with human feedback (2022) · cites this paper as Leike et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 149

  1. Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning , pp. 1-8, 2004.
  2. David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in-the-loop reinforcement learning. arXiv preprint arXiv:1701.04079 , 2017. · arXiv
  3. Bruce Abramson. The Expected-Outcome Model of Two-Player Games . PhD thesis, Columbia University, 1987.
  4. Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning , pp. 22-31, 2017.
  5. Riad Akrour, Marc Schoenauer, and Michèle Sebag. APRIL: Active preference learning-based reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pp. 116-131, 2012.
  6. Riad Akrour, Marc Schoenauer, Michèle Sebag, and Jean-Christophe Souplet. Programming by feedback. In International Conference on Machine Learning , pp. 1503-1511, 2014.
  7. Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. In Advances in Neural Information Processing Systems , pp. 1815-1824, 2017.
  8. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016. · arXiv
  9. Jacob Andreas, Anca Dragan, and Dan Klein. Translating neuralese. In Association for Computational Linguistics , pp. 232-242, 2017.
  10. Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 2166-2179, 2018.
  11. Anonymous. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations , 2019a. URL https://openreview.net/forum?id=r1lyTjAqYX . Under review.
  12. Anonymous. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations , 2019b. URL https://openreview.net/forum?id=HkezXnA9YX . Under review.
  13. Anonymous. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. In International Conference on Learning Representations , 2019c. URL https://openreview.net/forum? id=B1xhQhRcK7 . Under review.
  14. Stuart Armstrong. Motivated value selection for artificial agents. In AAAI Workshop: AI and Ethics , 2015.
  15. Stuart Armstrong and Benjamin Levinstein. Low impact artificial intelligences. arXiv preprint arXiv:1705.10720 , 2017. · arXiv
  16. Stuart Armstrong and Sören Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents. In Advances in Neural Information Processing Systems , 2018.
  17. Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approach . Cambridge University Press, 2009.
  18. Isaac Asimov. Runaround. Astounding Science Fiction , 29(1):94-103, 1942.
  19. Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 , 2018. · arXiv
  20. Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching YouTube. In Advances in Neural Information Processing Systems , 2018.
  21. James Babcock, János Kramár, and Roman Yampolskiy. The AGI containment problem. In Artificial General Intelligence , pp. 53-63, 2016.
  22. Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning to follow language instructions with adversarial reward induction. arXiv preprint arXiv:1806.01946 , 2018. · arXiv
  23. Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner's dilemma: Program equilibrium via provability logic. arXiv preprint arXiv:1401.5577 , 2014. · arXiv
  24. Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973 , 2018. · arXiv
  25. Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations , 2018.
  26. Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. arXiv preprint arXiv:1805.08328 , 2018. · arXiv
  27. Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018. · arXiv
  28. Seth D Baum. Social choice ethics in artificial intelligence. AI & Society , pp. 1-12, 2017.
  29. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013.
  30. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on International Conference on Machine Learning , pp. 1613-1622, 2015.
  31. Nick Bostrom. Ethical issues in advanced artificial intelligence. Science Fiction and Philosophy: From Time Travel to Superintelligence , pp. 277-284, 2003.
  32. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, 2014.
  33. Tom B Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, and Ian Goodfellow. Unrestricted adversarial examples. arXiv preprint arXiv:1809.08352 , 2018. · arXiv
  34. Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize via recursion. In International Conference on Learning Representations , 2017.
  35. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pp. 4299-4307, 2017.
  36. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575 , 2018. · arXiv
  37. Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. In Advances in Neural Information Processing Systems , pp. 271-278, 1993.
  38. Daniel Dewey. Learning what to value. In Artificial General Intelligence , pp. 309-314, 2011.
  39. Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265 , 2018a. · arXiv
  40. Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin. Score-based inverse reinforcement learning. In International Conference on Autonomous Agents & Multiagent Systems , pp. 457-465, 2016.
  41. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018. · arXiv
  42. Oren Etzioni and Daniel Weld. The first law of robotics (a call to arms). In AAAI , pp. 1042-1047, 1994.
  43. Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the preferences of ignorant, inconsistent agents. In AAAI , pp. 323-329, 2016.
  44. Owain Evans, Andreas Stuhlmüller, Chris Cundy, Ryan Carey, Zachary Kenton, Thomas McGrath, and Andrew Schreiber. Predicting human deliberative judgments with machine learning. Technical report, University of Oxford, 2018.
  45. Tom Everitt. Towards Safe Artificial General Intelligence . PhD thesis, Australian National University, 2018.
  46. Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel. In International Joint Conference on Artificial Intelligence , pp. 4705-4713, 2017.
  47. Tom Everitt, Gary Lea, and Marcus Hutter. AGI safety literature review. arXiv preprint arXiv:1805.01109 , 2018. · arXiv
  48. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning , pp. 49-58, 2016.
  49. Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine Learning , 89(1-2):123-156, 2012.
  50. Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems , pp. 3581-3590, 2017a.
  51. Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning , pp. 1183-1192, 2017b.
  52. Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.
  53. Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, and Jessica Taylor. Logical induction. arXiv preprint arXiv:1609.03543 , 2016. · arXiv
  54. Justin Gilmer, Ryan P Adams, Ian Goodfellow, David Andersen, and George E Dahl. Motivating the rules of the game for adversarial example research. arXiv preprint arXiv:1807.06732 , 2018. · arXiv
  55. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Artificial Intelligence and Statistics , pp. 315-323, 2011.
  56. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations , 2015. URL http://arxiv.org/abs/1412. 6572 .
  57. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning . MIT Press, 2016.
  58. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International Conference on Machine Learning , 2013.
  59. Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems , pp. 2348-2356, 2011.
  60. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwi´ nska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature , 538(7626):471, 2016.
  61. Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Atari agents. In International Conference on Machine Learning , pp. 1792-1801, 2018.
  62. Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems , pp. 2625-2633, 2013.
  63. Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems , pp. 3909-3917, 2016.
  64. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems , pp. 6765-6774, 2017b.
  65. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017.
  66. José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning , pp. 1861-1869, 2015.
  67. Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, John Agapiou, Joel Z Leibo, and Audrunas Gruslys. Deep Q-learning from demonstrations. In AAAI , 2018.
  68. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems , pp. 6626-6637, 2017.
  69. Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representaitons , 2018.
  70. Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284 , 2017. · arXiv
  71. Marcus Hutter. Universal artificial intelligence . Springer, 2005.
  72. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems , 2018.
  73. Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv preprint arXiv:1807.01281 , 2018. · arXiv
  74. Richard Karp. Reducibility among combinatorial problems. In Complexity of Computer Computations , pp. 85-103. Springer, 1972.
  75. Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems , pp. 5574-5584, 2017.
  76. Been Kim, Justin Gilmer, Fernanda Viegas, Ulfar Erlingsson, and Martin Wattenberg. TCAV: Relative concept importance testing with linear concept activation vectors. arXiv preprint arXiv:1711.11279 , 2017. · arXiv
  77. Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems , pp. 2575-2583, 2015.
  78. William Bradley Knox. Learning from human-generated reward . PhD thesis, University of Texas, 2012.
  79. William Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In International Conference on Knowledge Capture , pp. 9-16, 2009.
  80. Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In European Conference on Machine Learning , pp. 282-293, 2006.
  81. Victoria Krakovna, Laurent Orseau, Miljan Martic, and Shane Legg. Measuring and avoiding side effects using relative reachability. arXiv preprint arXiv:1806.01186 , 2018. · arXiv
  82. Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627 , 2018. · arXiv
  83. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems , pp. 1097-1105, 2012.
  84. David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observing rewards at a cost. In Future of Interactive Learning Machines, NIPS Workshop , 2016.
  85. Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning , pp. 2801-2809, 2018.
  86. Isaac Lage, Andrew Slavin Ross, Been Kim, Samuel J Gershman, and Finale Doshi-Velez. Human-in-the-loop interpretability prior. arXiv preprint arXiv:1805.11571 , 2018. · arXiv
  87. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , pp. 6402-6413, 2017.
  88. Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems , pp. 396-404, 1990.
  89. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436, 2015.
  90. Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines , 17(4):391-444, 2007.
  91. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Belson, David M Bryson, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie Forrest, Antoine Frénoy, Christian Gagné, Leni Le Goff, Laura M Grabowski, Babak Hodjat, Frank Hutter, Laurent Keller, Carole Knibbe, Peter Krcah, Richard E Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Miikkulainen, Sara Mitri, David E Moriarty, Jean-Baptiste Mouret, Anh Nguyen, Charles Ofria, Marc Parizeau, David Parsons, Robert T Pennock, William F Punch, Thomas S Ray, Marc Schoenauer, Eric Shulte, Karl Sims, Kenneth O Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Westley Weimer, Richard Watson, and Jason Yosinski. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. arXiv preprint arXiv:1803.03453 , 2018. · arXiv
  92. Jan Leike and Marcus Hutter. Bad universal priors and notions of optimality. In Conference on Learning Theory , pp. 1244-1259, 2015.
  93. Jan Leike, Jessica Taylor, and Benya Fallenstein. A formal solution to the grain of truth problem. In Conference on Uncertainty in Artificial Intelligence , pp. 427-436, 2016.
  94. Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv preprint arXiv:1711.09883 , 2017. · arXiv
  95. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015. · arXiv
  96. James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning , pp. 2285-2294, 2017.
  97. Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? In International Joint Conference on Artificial Intelligence , pp. 4754-4760, 2017.
  98. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529, 2015.
  99. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pp. 1928-1937, 2016.
  100. Brendan Mulvaney. Red teams. Marine Corps Gazette , 96(7):63, 2012.
  101. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning , pp. 807-814, 2010.
  102. Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning , pp. 663-670, 2000.
  103. Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning , pp. 278-287, 1999.
  104. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill , 2018. https://distill.pub/2018/ building-blocks .
  105. Stephen Omohundro. The basic AI drives. In Artificial General Intelligence , pp. 483-492, 2008.
  106. Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In Conference on Uncertainty in Artificial Intelligence , pp. 557-566, 2016.
  107. Laurent Orseau and Mark Ring. Space-time embedded intelligence. In Artificial General Intelligence , pp. 209-218, 2012.
  108. Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation , 3(1):88-97, 1991.
  109. Mahendra Prasad. Social choice and the value alignment problem. In Artificial Intelligence Safety and Security , pp. 291-314. Chapman and Hall/CRC, 2018.
  110. Mark O Riedl and Brent Harrison. Using stories to teach human values to artificial agents. In AAAI Workshop: AI, Ethics, and Society , 2016.
  111. Mark Ring and Laurent Orseau. Delusion, survival, and intelligent agents. In Artificial General Intelligence , pp. 11-20, 2011.
  112. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. In Parallel distributed processing: Explorations in the Microstructure of Cognition , volume 1. MIT Press, 1986.
  113. Stuart Russell. Learning agents for uncertain environments. In Conference on Computational Learning Theory , pp. 101-103, 1998.
  114. Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI Magazine , 36(4):105-114, 2015.
  115. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems , pp. 2234-2242, 2016.
  116. Paul A Samuelson. A note on the pure theory of consumer's behaviour. Economica , 5(17):61-71, 1938.
  117. William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In International Conference on Autonomous Agents and MultiAgent Systems , pp. 2067-2069, 2018.
  118. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning , pp. 1889-1897, 2015.
  119. Sebastian Schulze and Owain Evans. Active reinforcement learning with monte-carlo tree search. arXiv preprint arXiv:1803.04926 , 2018. · arXiv
  120. Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning , 6(1):1-114, 2012.
  121. Alireza Shafaei, Mark Schmidt, and James J Little. Does your model know the digit 6 is not a cat? a less biased evaluation of 'outlier' detectors. arXiv preprint arXiv:1809.04729 , 2018. · arXiv
  122. Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. On a formal model of safe and scalable selfdriving cars. arXiv preprint arXiv:1708.06374 , 2017. · arXiv
  123. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484, 2016.
  124. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Graepel Thore, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature , 550(7676):354, 2017.
  125. Nate Soares and Benya Fallenstein. Agent foundations for aligning machine intelligence with human interests: a technical research agenda. In The Technological Singularity , pp. 103-125. Springer, 2017.
  126. Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In AAAI Workshop on Artificial Intelligence and Ethics , 2015.
  127. Thilo Stadelmann, Mohammadreza Amirian, Ismail Arabaci, Marek Arnold, Gilbert François Duivesteijn, Ismail Elezi, Melanie Geiger, Stefan Lörwald, Benjamin Bruno Meier, Katharina Rombach, and Lukas Tuggener. Deep learning in the wild. arXiv preprint arXiv:1807.04950 , 2018. · arXiv
  128. Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. In International Conference on Learning Representations , 2017.
  129. Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction . MIT press, 2nd edition, 2018.
  130. Richard Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 112(1-2):181-211, 1999.
  131. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. · arXiv
  132. Andrea L Thomaz and Cynthia Breazeal. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence , 172(6-7):716-737, 2008.
  133. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Intelligent Robots and Systems , pp. 23-30, 2017.
  134. Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems , 2018.
  135. Hsiao-Yu Fish Tung, Adam W Harley, Liang-Kang Huang, and Katerina Fragkiadaki. Reward learning from narrated demonstrations. In Computer Vision and Pattern Recognition , pp. 7004-7013, 2018.
  136. Jonathan Uesato, Brendan O'Donoghue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning , pp. 5025-5034, 2018.
  137. Joel Veness, Tor Lattimore, Avishkar Bhoopchand, Agnieszka Grabska-Barwinska, Christopher Mattern, and Peter Toth. Online learning with gated linear networks. arXiv preprint arXiv:1712.01897 , 2017. · arXiv
  138. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161 , 2017. · arXiv
  139. Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. arXiv preprint arXiv:1709.10163 , 2017. · arXiv
  140. Christopher Watkins and Peter Dayan. Q-learning. Machine Learning , 8(3-4):279-292, 1992.
  141. Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8(3-4):229-256, 1992.
  142. Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18(1):4945-4990, 2017.
  143. Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems , 2018.
  144. Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 , 2015. · arXiv
  145. Roman Yampolskiy. Leakproofing the singularity: Artificial intelligence confinement problem. Journal of Consciousness Studies , 2012.
  146. Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding DQNs. In International Conference on Machine Learning , pp. 1899-1908, 2016.
  147. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017.
  148. Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893 , 2018a. · arXiv
  149. Shun Zhang, Edmund H Durfee, and Satinder Singh. Minimax-regret querying on side effects for safe optimality in factored Markov decision processes. In International Joint Conference on Artificial Intelligence , pp. 4867-4873, 2018b.

Cited by

Sorted by citations

Authors
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1811.07871
arXiv
1811.07871
OpenAlex
W2901707424
Cited by
121
References
15
Citations by year
2026: 4 · 2025: 14 · 2024: 18 · 2023: 16 · 2022: 9 · 2021: 18 · 2020: 24 · 2019: 18
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
agent alignment, reward modeling, reward function learning, reinforcement learning, human-robot interaction, user intention, scalability, trust in AI agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review