The DOI matches its DataCite record.
Repeated Inverse Reinforcement Learning
Open accessAbstract
We introduce a novel repeated Inverse Reinforcement Learning problem: the agent has to act on behalf of a human in a sequence of tasks and wishes to minimize the number of tasks that it surprises the human by acting suboptimally with respect to how the human would have acted. Each time the human is surprised, the agent is provided a demonstration of the desired behavior by the human. We formalize this problem, including how the sequence of tasks is chosen, in a few different ways and provide some foundational results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reward is under-determined by behavior without intervention.
“In other words, reward is under-determined by behavior (Cao et al., 2021) without intervention (Amin et al., 2017).”
From Agency Is Frame-Dependent (2025) · cites this paper as Amin et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 20
- Pieter Abbeel and Andrew Y Ng. Apprenticeship Learning via Inverse Reinforcement Learning. In Proceedings of the 21st International Conference on Machine learning , page 1. ACM, 2004.
- Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y Ng. An application of reinforcement learning to aerobatic helicopter flight. Advances in neural information processing systems , 19:1, 2007.
- Kareem Amin and Satinder Singh. Towards resolving unidentifiability in inverse reinforcement learning. arXiv preprint arXiv:1601.06569 , 2016. · arXiv
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 , 2016. · arXiv
- Nick Bostrom. Ethical issues in advanced artificial intelligence. Science Fiction and Philosophy: From Time Travel to Superintelligence , pages 277-284, 2003.
- Urszula Chajewska, Daphne Koller, and Ronald Parr. Making rational decisions using adaptive utility elicitation. In AAAI/IAAI , pages 363-369, 2000.
- Adam Coates, Pieter Abbeel, and Andrew Y Ng. Learning for control from multiple demonstrations. In Proceedings of the 25th international conference on Machine learning , pages 144-151. ACM, 2008.
- Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combinatorial optimization , volume 2. Springer Science & Business Media, 2012.
- Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems , pages 39093917, 2016.
- Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning , pages 663-670, 2000.
- Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd International Conference on Machine Learning , pages 729-736. ACM, 2006.
- Kevin Regan and Craig Boutilier. Regret-based reward elicitation for markov decision processes. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , pages 444-451. AUAI Press, 2009.
- Kevin Regan and Craig Boutilier. Robust policy computation in reward-uncertain mdps using nondominated policies. In AAAI , 2010.
- Kevin Regan and Craig Boutilier. Eliciting additive reward functions for markov decision processes. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence , volume 22, page 2159, 2011.
- Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases , pages 34-48. Springer, 2011.
- Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI Magazine , 36(4):105-114, 2015.
- Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107-194, 2011.
- Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems , pages 1449-1456, 2007.
- John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior (60th Anniversary Commemorative Edition) . Princeton university press, 2007.
- Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI , pages 1433-1438, 2008.
Cited by
Sorted by citations
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Open Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93
Abel, D., Barreto, A., Bowling, M., Dabney, W., Dong, S., Hansen, S., Harutyunyan, A., Khetarpal, K., Lyle, C., Pascanu, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04403
“In other words, reward is under-determined by behavior (Cao et al., 2021) without (Amin et al., 2017).”
Self-citation
Metadata
- Authors
- Amin, K., Jiang, N., & Singh, S.
- Affiliations
- Google (United States) (US)
- University of Michigan (US)
- ORCID
- Singh, Satinder
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 30
- Pages
- 1815-1824
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1705.05427
- OpenAlex
- W2621205314
- Cited by
- 40
- References
- 0
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 3 · 2022: 1 · 2021: 6 · 2020: 9 · 2019: 8 · 2018: 7 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Funders
- National Science Foundation (1319365)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- inverse reinforcement learning, repeated games, human-robot interaction, human demonstrations, suboptimal behavior, human-in-the-loop learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
