Cooperative Inverse Reinforcement Learning
Open accessAbstract
For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial-information game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Assistance games, such as cooperative inverse RL (CIRL), provide a framework to analyze algorithms like RLHF.
“Assistance games, such as cooperative inverse RL (CIRL) (Hadfield-Menell et al., 2016), provide a framework to analyze algorithms like RLHF.”
From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Hadfield-Menell et al., 2016
The Introduction (p. 2) defines CIRL as a cooperative human–robot reward-learning game, and the Conclusion (p. 8) says it provides a framework for designing HRI algorithms and analyzing incentives in reward learning. Our copy does not discuss RLHF or establish that CIRL can be used to analyze RLHF specifically. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Theory of the firm: Managerial behavior, agency costs and ownership structure
Jensen, M. C., & Meckling, W. H. (1976) · Journal of Financial Economics · doi:10.1016/0304-405x(76)90026-x · Cited by 71,491 · see citing works
Multitask principal – agent analyses: Incentive contracts, asset ownership, and job design
Holmstrom, B., & Milgrom, P. (1991) · The Journal of Law, Economics, and Organization · doi:10.1093/jleo/7.special_issue.24 · Cited by 6,046 · see citing works
Aggregation and Linearity in the Provision of Intertemporal Incentives
Holmstrom, B. R., & Milgrom, P. (1987) · Econometrica · doi:10.2307/1913238 · Cited by 2,870 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015430 · Cited by 2,855 · see citing works
Self-citationSuperintelligence: paths, dangers, strategies
(2015) · Choice Reviews Online · doi:10.5860/choice.187536 · Cited by 1,977 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R., & Sondik, E. (1973) · Operational Research · doi:10.1287/opre.21.5.1071 · Cited by 1,787 · see citing works
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · see citing works
On the folly of rewarding A, while hoping for B
Kerr, S. (1975) · Academy of Management Journal · doi:10.2307/255378 · Cited by 1,585 · see citing works
An investigation of the Therac-25 accidents
Leveson, N. G., & Turner, C. S. (1993) · Computer · doi:10.1109/mc.1993.274940 · Cited by 1,245 · see citing works
Ratliff, N., Bagnell, J. A., & Zinkevich, M. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143936 · Cited by 617 · see citing works
Goldman, S., & Kearns, M. (1991) · Annual Conference Computational Learning Theory · doi:10.1006/jcss.1995.1003 · Cited by 419 · see citing works
Wiener, N. (1960) · Science · doi:10.1126/science.131.3410.1355 · Cited by 367 · cited 2× · see citing works
“a mechanical agency with whose operation we cannot interfere effectively :::we had better be quite sure that the purpose put into the machine is the purpose which we really desire.” So wrote Norbert Wiener (1960) in one of the earliest explanations of the problems that arise when autonomous systems operate with object…”
BackgroundLearning agents for uncertain environments (extended abstract)
Russell, S. (1998) · Conference on Learning Theory · doi:10.1145/279943.279964 · Cited by 298 · see citing works
Self-citationSequential Optimality and Coordination in Multiagent Systems
Boutilier, C. E. (1999) · Cited by 294 · see citing works
Decentralized Stochastic Control with Partial History Sharing: A Common Information Approach
Nayyar, A., Mahajan, A., & Teneketzis, D. (2013) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2013.2239000 · Cited by 279 · see citing works
The Complexity of Decentralized Control of Markov Decision ProcessesPreprint
Bernstein, D. S., Zilberstein, S., & Immerman, N. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3836 · Cited by 238 · see citing works
A Game-Theoretic Approach to Generating Spatial Descriptions
Golland, D., Liang, P., & Klein, D. (2010) · Cited by 132 · see citing works
Dragan, A. D., & Srinivasa, S. S. (2013) · Robotics: Science and Systems · doi:10.15607/rss.2013.ix.024 · Cited by 117 · see citing works
Self-citationGibbons, R. S. (1998) · National Bureau of Economic Research · doi:10.3386/w6695 · Cited by 99 · see citing works
Dragan, A. D., & Srinivasa, S. S. (2018) · doi:10.1184/r1/6554969.v1 · Cited by 97 · see citing works
Self-citationMulti-Agent Inverse Reinforcement Learning
Natarajan, S., Kunapuli, G., Judah, K., Tadepalli, P. V., Kristian Kersting, B., & SHAVLIK, J. W. (2010) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2010.65 · Cited by 83 · see citing works
Algorithmic and Human Teaching of Sequential Decision Tasks
Çakmak, M., & Lopes, M. C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8333 · Cited by 77 · see citing works
Learning Binary Relations and Total Orders
Goldman, S. A., Rivest, R. L., & Schapire, R. E. (1993) · SIAM Journal on Computing · doi:10.1137/0222062 · Cited by 64 · see citing works
A Decision-Theoretic Model of Assistance
Fern, A., Natarajan, S., Judah, K., & Tadepalli, P. V. (2014) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4213 · Cited by 38 · see citing works
Bayesian inverse reinforcement learning
Ramachandran, D., & Eyal Amir, A. (2007) · Cited by 26 · see citing works
Recent Developments in Algorithmic Teaching
Balbach, F. J., & Zeugmann, T. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-00982-2_1
Flagged first, then by citations
Research works printed but not linked: 2
- Russell, S. and Norvig, P. Artificial Intelligence . Pearson, 2010.
- Waugh, K, Ziebart, B, and Bagnell, J. Computational rationalization: The inverse equilibrium problem. In ICML , 2011.
Cited by
Sorted by citations
Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2977374 · Cited by 1,306 · see citing works
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · see citing works
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · see citing works
Recent Advances in Robot Learning from Demonstration
Ravichandar, H., Polydoros, A. S., Chernova, S., & Billard, A. (2020) · Annu. Rev. Control. Robotics Auton. Syst. · doi:10.1146/annurev-control-100819-063206 · Cited by 947 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
An algorithmic perspective on imitation learning
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., & Peters, J. (2018) · Foundations and Trends in Robotics · doi:10.1561/2300000053 · Cited by 405 · see citing works
Self-citationAn Algorithmic Perspective on Imitation Learning
Osa, T., Pajarinen, J., Neumann, G. M., Bagnell, J. A., Abbeel, P., & Peters, J. (2018) · now publishers, Inc. eBooks · doi:10.1561/9781680834116 · Cited by 387 · see citing works
Self-citationA Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings
Mukherjee, D., Gupta, K., Chang, L., & Najjaran, H. (2021) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2021.102231 · Cited by 349 · see citing works
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, Y.-X., & Knoll, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3457538 · Cited by 341 · see citing works
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. G. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 405 · see citing works
Russell, S. (2020) · doi:10.1093/oso/9780190905033.003.0012 · Cited by 289 · see citing works
Self-citationDu, W., & Ding, S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09938-y · Cited by 278 · see citing works
Machine Theory of MindPreprint
Rabinowitz, N. C., Perbet, F., Song, H. F., Zhang, C., Eslami, S. M. A., & Botvinick, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07740 · Cited by 259 · see citing works
Vallor, S. (2024) · doi:10.1093/oso/9780197759066.001.0001 · Cited by 252 · see citing works
Li, C. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9781351006620-6 · Cited by 251 · see citing works
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
Zare, M., Kebria, P. M., Khosravi, A., & Nahavandi, S. (2024) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2024.3395626 · Cited by 230 · see citing works
Improved Trust in Human-Robot Collaboration With ChatGPT
Ye, Y., You, H., & Du, J. (2023) · IEEE Access · doi:10.1109/access.2023.3282111 · Cited by 213 · see citing works
Theory of mind as inverse reinforcement learning
Jara‐Ettinger, J. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.04.010 · Cited by 203 · see citing works
An Analysis of the Interaction Between Intelligent Software Agents and Human Users
Burr, C., Cristianini, N., & Ladyman, J. (2018) · Minds and Machines · doi:10.1007/s11023-018-9479-0 · Cited by 184 · see citing works
A review on reinforcement learning for contact-rich robotic manipulation tasks
Elguea-Aguinaco, Í., Serrano-Muñoz, A., Chrysostomou, D., Inziarte-Hidalgo, I., Bøgh, S., & Arana-Arexolaleiba, N. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102517 · Cited by 168 · see citing works
Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., & Legg, S. (2017) · arXiv · doi:10.48550/arxiv.1711.09883 · Cited by 240 · see citing works
Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge
Wang, P., & Chan, C. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317735 · Cited by 151 · see citing works
Kim, S. K., Kirchner, E. A., Stefes, A., & Kirchner, F. (2017) · Scientific Reports · doi:10.1038/s41598-017-17682-7 · Cited by 147 · see citing works
From inverse optimal control to inverse reinforcement learning: A historical review
Azar, N. A., Shahmansoorian, A., & Davoudi, M. (2020) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2020.06.001 · Cited by 127 · see citing works
Modeling Others using Oneself in Multi-Agent Reinforcement LearningPreprint
Răileanu, R., Denton, E., Szlam, A. D., & Fergus, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09640 · Cited by 126 · see citing works
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., & Dragan, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02827 · Cited by 125 · see citing works
Self-citationHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/32 · Cited by 119 · see citing works
Self-citationHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1611.08219 · Cited by 119
Self-citationA Deep Reinforcement Learning Framework for Fast Charging of Li-Ion Batteries
Park, S., Pozzi, A., Whitmeyer, M., Perez, H. E., Kandel, A., Kim, G., Choi, Y., Joe, W. T., Raimondo, D. M., & Moura, S. J. (2022) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2022.3140316 · Cited by 113 · see citing works
Perspectives on Digital Humanism
Werthner, H., Prem, E., Lee, E. A., & Ghezzi, C. (2021) · Springer eBooks · doi:10.1007/978-3-030-86144-5 · Cited by 117 · see citing works
Game-Theoretic Modeling of Human Adaptation in Human-Robot Collaboration
Nikolaidis, S., Nath, S., Procaccia, A. D., & Srinivasa, S. (2017) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2909824.3020253 · Cited by 102 · see citing works
Learning to Teach in Cooperative Multiagent Reinforcement Learning
Omidshafiei, S., Kim, D.-K., Liu, M., Tesauro, G. J., Riemer, M., Amato, C., Campbell, M. S., & How, J. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016128 · Cited by 23 · see citing works
Artificial Intelligence and the Problem of Control
Russell, S. (2021) · Springer eBooks · doi:10.1007/978-3-030-86144-5_3 · Cited by 100 · see citing works
Self-citationA Survey of Mental Modeling Techniques in Human–Robot Teaming
Tabrez, A., Luebbers, M. B., & Hayes, B. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00019-0 · Cited by 98 · see citing works
Unsolved Problems in ML Safety
Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2109.13916 · Cited by 95 · see citing works
Inverse Reinforcement Q-Learning Through Expert Imitation for Discrete-Time Systems
Xue, W., Lian, B., Fan, J., Kolaric, P., Chai, T., & Lewis, F. L. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3106635 · Cited by 95 · see citing works
(2021) · Journal of Machine Learning Research · Cited by 224 · see citing works
Open Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93 · see citing works
Understanding Human Intelligence through Human Limitations
Griffiths, T. L. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.001 · Cited by 88 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“Assistance games, such as cooperative inverse RL (CIRL) [Hadfield-Menell et al., 2016], provide a framework to analyze algorithms like RLHF.”
MethodSelf-citationValiente, R., Toghi, B., Pedarsani, R., & Fallah, Y. P. (2022) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2022.3172981 · Cited by 77 · see citing works
Reinforcement Learning with a Corrupted Reward Channel
Everitt, T., Krakovna, V., Orseau, L., & Legg, S. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/656 · Cited by 74 · see citing works
Trustworthy human-AI partnerships
Ramchurn, S. D., Stein, S., & Jennings, N. R. (2021) · iScience · doi:10.1016/j.isci.2021.102891 · Cited by 73 · see citing works
Showing versus Doing: Teaching by DemonstrationPreprint
Ho, M. K., Littman, M. L., MacGlashan, J., Cushman, F. A., & Austerweil, J. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14004 · Cited by 71 · see citing works
Learning a commonsense moral theory.
Kleiman-Weiner, M., Saxe, R., & Tenenbaum, J. (2017) · Cognition · doi:10.1016/j.cognition.2017.03.005 · Cited by 84 · see citing works
Rakhsha, A., Radanović, G., Devidze, R., Zhu, X., & Singla, A. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.12909 · Cited by 68 · see citing works
Inferring Human Intent and Predicting Human Action in Human–Robot Collaboration
Hoffman, G., Bhattacharjee, T., & Nikolaidis, S. (2023) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-071223-105834 · Cited by 66 · see citing works
Find the Gap: AI, Responsible Agency and Vulnerability
Vallor, S., & Vierkant, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09674-0 · Cited by 66 · see citing works
Hard choices in artificial intelligence
Dobbe, R., Krendl Gilbert, T., & Mintz, Y. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103555 · Cited by 65 · see citing works
Langley, C., Cirstea, B. I., Fabio Cuzzolin, A., & Sahakian, B. J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.778852 · Cited by 64 · see citing works
Flexible human–robot cooperation models for assisted shop-floor tasks
Darvish, K., Wanderlingh, F., Bruno, B., Simetti, E., Mastrogiovanni, F., & Casalino, G. (2018) · Mechatronics · doi:10.1016/j.mechatronics.2018.03.006 · Cited by 62 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3377445 · Cited by 62 · see citing works
Zhang, H., Zhang, Y., Wang, Z., Zhang, S., Li, H., & Chen, M. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2023.04.005 · Cited by 59 · see citing works
A goal-centric outlook on learning
Molinaro, G., & Collins, A. G. E. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.08.011 · Cited by 63 · see citing works
Alignment for Advanced Machine Learning Systems
Taylor, J., Yudkowsky, E., LaVictoire, P., & Critch, A. (2020) · doi:10.1093/oso/9780190905033.003.0013 · Cited by 59 · see citing works
Robot learning from demonstration for path planning: A review
Xie, Z., Zhang, Q., Jiang, Z., & Liu, H. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1648-4 · Cited by 58 · see citing works
LUCIDGames: Online Unscented Inverse Dynamic Games for Adaptive Trajectory Prediction and Planning
Cleac’h, S. L., Schwager, M., & Manchester, Z. R. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3074880 · Cited by 58 · see citing works
Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography
Gschwindt, M., Camci, E., Bonatti, R., Wang, W., Kayacan, E., & Scherer, S. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967592 · Cited by 58 · see citing works
Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions
Mehr, N., Wang, M., Bhatt, M., & Schwager, M. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3232300 · Cited by 57 · see citing works
When code isn’t law: rethinking regulation for artificial intelligence
Judge, B., Nitzberg, M., & Russell, S. J. (2024) · Policy & Society · doi:10.1093/polsoc/puae020 · Cited by 60 · see citing works
Self-citationAwad, E., Levine, S., Anderson, M., Anderson, S., Conitzer, V., Crockett, M. J., Everett, J. A. C., Evgeniou, T., Alison Gopnik, C., Jamison, J., et al. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.02.009 · Cited by 58 · see citing works
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications
Brown, D. S., & Niekum, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017749 · Cited by 12 · see citing works
The Challenge of Value Alignment: From Fairer Algorithms to AI Safety
Gabriel, I., & Ghazavi, V. (2022) · Oxford Handbook of Digital Ethics · doi:10.1093/oxfordhb/9780198857815.013.18 · Cited by 52 · see citing works
Personalizing Mobile Fitness Apps using Reinforcement Learning.
Zhou, M., Mintz, Y. D., Fukuoka, Y., Goldberg, K. Y., Flowers, E., Kaminsky, P., Castillejo, A., & Aswani, A. J. (2018) · PubMed · Cited by 52 · see citing works
Foundations of explanations as model reconciliation
Sreedharan, S., Chakraborti, T., & Subbarao Kambhampati, B. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103558 · Cited by 51 · see citing works
Agentic AI: The age of reasoning—A review
Nisa, U., Shirazi, M., Saip, M. A., & Pozi, M. S. M. (2025) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2025.08.003 · Cited by 71 · see citing works
Continuous-time inverse quadratic optimal control problem
Li, Y., Yao, Y., & Xiaoming Hu, A. (2020) · Automatica · doi:10.1016/j.automatica.2020.108977 · Cited by 50 · see citing works
Franceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 52 · see citing works
Everitt, T., Hutter, M., Kumar, R., & Krakovna, V. (2021) · Synthese · doi:10.1007/s11229-021-03141-4 · Cited by 80 · see citing works
Safe Imitation Learning via Fast Bayesian Reward Inference from PreferencesPreprint
Brown, D. S., Coleman, R., Srinivasan, R., & Niekum, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09089 · Cited by 51 · see citing works
Clustering Trust Dynamics in a Human-Robot Sequential Decision-Making Task
Bhat, S., Lyons, J. B., Shi, C., & Yang, X. J. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3188902 · Cited by 46 · see citing works
AI Safety Needs Social Scientists
Irving, G., & Askell, A. (2019) · Distill · doi:10.23915/distill.00014 · Cited by 46 · see citing works
The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail
Bowman, S. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.516 · Cited by 45 · see citing works
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback
Spencer, J. C., Choudhury, S., Barnes, M., Schmittle, M., Chiang, M., Ramadge, P. J., & Srinivasa, S. S. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.055 · Cited by 44 · see citing works
Who's Learning? Using Demographics in EDM Research
Paquette, L., Ocumpaugh, J. L., Li, Z., Andres, A., & Baker, R. S. (2020) · Journal of Educational Data Mining · doi:10.5281/zenodo.4143611 · Cited by 46 · see citing works
Butz, M. V. (2021) · KI - Künstliche Intelligenz · doi:10.1007/s13218-021-00705-x · Cited by 42 · see citing works
Ho, M. K., & Griffiths, T. L. (2021) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-042920-015547 · Cited by 41 · see citing works
Quantum Computing and AI: The Synergy of Two Revolutionary Technologies
Ahmadi, A. A. (2023) · Asian Journal of Electrical Sciences · doi:10.51983/ajes-2023.12.2.4118 · Cited by 44 · see citing works
Handling Model Uncertainty and Multiplicity in Explanations via Model Reconciliation
Sreedharan, S., Chakraborti, T., & Subbarao Kambhampati, B. (2018) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v28i1.13930 · Cited by 42 · see citing works
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Gao, X.-F., Gong, R., Zhao, Y.-Z., Wang, S., Shu, T., & Zhu, S.-C. (2020) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/ro-man47096.2020.9223595 · Cited by 48 · see citing works
Multiagent Graphical Games With Inverse Reinforcement Learning
Donge, V. S., Lian, B., Lewis, F. L., & Davoudi, A. A. (2023) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2022.3210856 · Cited by 39 · see citing works
Multi-agent Inverse Reinforcement Learning for Certain General-sum Stochastic Games
Lin, X., Adams, S. C., & Beling, P. A. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11541 · Cited by 39 · see citing works
A Survey of Opponent Modeling in Adversarial Domains
Nashed, S. B., & Shlomo Zilberstein, B. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12889 · Cited by 38 · see citing works
Physical interaction as communication: Learning robot objectives online from human corrections
Losey, D. P., Bajcsy, A. V., O’Malley, M. K., & Dragan, A. D. (2021) · The International Journal of Robotics Research · doi:10.1177/02783649211050958 · Cited by 37 · see citing works
Self-citationInteractive Teaching Algorithms for Inverse Reinforcement Learning
Kamalaruban, P., Devidze, R., Cevher, V., & Singla, A. K. (2019) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 37 · see citing works
Conservative Agency via Attainable Utility Preservation
Turner, A. M., Hadfield-Menell, D., & Tadepalli, P. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375851 · Cited by 36 · see citing works
Self-citationLai, K.-H., Zha, D., Li, Y., & Hu, X. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/435 · Cited by 40 · see citing works
Human-in-the-Loop Imitation Learning using Remote Teleoperation
Mandlekar, A., Xu, D., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S. (2020) · arXiv · doi:10.48550/arxiv.2012.06733 · Cited by 35 · see citing works
Decision-Oriented Dialogue for Human-AI Collaboration
Lin, J., Tomlin, N., Andreas, J., & Eisner, J. M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00679 · Cited by 35 · see citing works
Beyond Preferences in AI Alignment
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 85 · see citing works
Task Learning, Intent Prediction, and Adaptive Blended Shared Control With Application to Excavators
Jin, Z., Pagilla, P. R., Maske, H., & Chowdhary, G. V. (2020) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2019.2959536 · Cited by 35 · see citing works
AI ethical biases: normative and information systems development conceptual framework
Chowdhury, T., & Oredo, J. O. (2022) · Journal of Decision System · doi:10.1080/12460125.2022.2062849 · Cited by 38 · see citing works
AGENT: A Benchmark for Core Psychological Reasoning
Shu, T., Bhandwaldar, A., Gan, C., Smith, K. A., Liu, S., Gutfreund, D., Spelke, E., Tenenbaum, J. B., & Ullman, T. D. (2021) · arXiv · doi:10.48550/arxiv.2102.12321 · Cited by 34 · see citing works
Algorithmic Equity: A Framework for Social Applications
Osoba, O., Boudreaux, B., Saunders, J. M., Irwin, J., Mueller, P. A., & Cherney, S. (2019) · RAND Corporation eBooks · doi:10.7249/rr2708 · Cited by 35 · see citing works
Game-Theoretic Planning for Risk-Aware Interactive Agents
Wang, M., Mehr, N., Gaidon, A., & Schwager, M. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341137 · Cited by 33 · see citing works
SRL-ORCA: A Socially Aware Multi-Agent Mapless Navigation Algorithm in Complex Dynamic Scenes
Qin, J., Qin, J., Qiu, J., Liu, Q., Li, M., & Ma, Q. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3331621 · Cited by 33 · see citing works
Instilling moral value alignment by means of multi-objective reinforcement learning
Rodríguez-Soto, M., Serramia, M., López-Sánchez, M., & Rodríguez-Aguilar, J. A. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09635-0 · Cited by 32 · see citing works
Bridging adaptive management and reinforcement learning for more robust decisions
Chapman, M. S., Xu, L., Lapeyrolerie, M., & Boettiger, C. (2023) · Philosophical Transactions of the Royal Society of London. Biological Sciences · doi:10.1098/rstb.2022.0195 · Cited by 31 · see citing works
Bobu, A., Bajcsy, A., Fisac, J. F., Deglurkar, S., & Dragan, A. D. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.2971415 · Cited by 33 · see citing works
Self-citationChallenges of Aligning Artificial Intelligence with Human Values
Sutrop, M. (2020) · Acta Baltica Historiae et Philosophiae Scientiarum · doi:10.11590/abhps.2020.2.04 · Cited by 31 · see citing works
- Authors
- Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- Volume
- 29
- Pages
- 3909-3917
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1606.03137
- OpenAlex
- W2410842990
- Cited by
- 413
- References
- 25
- Citations by year
- 2026: 1 · 2025: 18 · 2024: 59 · 2023: 56 · 2022: 37 · 2021: 66 · 2020: 80 · 2019: 41 · 2018: 27 · 2017: 25 · 2016: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- value alignment, human-robot interaction, partially observable Markov decision processes, active teaching methods, active learning (machine learning), communicative action, robot learning from humans, reward function learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
