Towards Resolving Unidentifiability in Inverse Reinforcement Learning
Open accessAbstract
We consider a setting for Inverse Reinforcement Learning (IRL) where the learner is extended with the ability to actively select multiple environments, observing an agent's behavior on each environment. We first demonstrate that if the learner can experiment with any transition dynamics on some fixed set of states and actions, then there exists an algorithm that reconstructs the agent's reward function to the fullest extent theoretically possible, and that requires only a small (logarithmic) number of experiments. We contrast this result to what is known about IRL in single fixed environments, namely that the true reward function is fundamentally unidentifiable. We then extend this setting to the more realistic case where the learner may not select any transition dynamic, but rather is restricted to some fixed set of environments that it may try. We connect the problem of maximizing the information derived from experiments to submodular function maximization and demonstrate that a greedy algorithm is near optimal (up to logarithmic factors). Finally, we empirically validate our algorithm on an environment inspired by behavioral psychology.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To fully determine the environment transition function, the optimal policy must be known for multiple goals.
“and likewise we find that to fully determine the environment transition function we must know the optimal policy for multiple goals.”
From General agents contain world models (2025) · cites this paper as Amin & Singh, 2016
In Setting and Preliminaries (p. 2), the paper defines transition dynamics as part of the environments selected by the learner and says the unknown quantity to estimate is the reward function. In Omnipotent Identification Algorithm (p. 5), it says repeated experiments can identify that reward function by varying environments, not determine a transition function from optimal policies for multiple goals. (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 19
- [1][1] P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng. An application of reinforcement learning to aerobatic helicopter flight. Advances in neural information processing systems , 19:1, 2007. -
- [2][2] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1. ACM, 2004. -
- [3][3] U. Chajewska, D. Koller, and R. Parr. Making rational decisions using adaptive utility elicitation. In AAAI/IAAI , pages 363-369, 2000. -
- [4][4] A. Coates, P. Abbeel, and A. Y. Ng. Learning for control from multiple demonstrations. In Proceedings of the 25th international conference on Machine learning , pages 144-151. ACM, 2008. -
- [5][5] A. Coates, P. Abbeel, and A. Y. Ng. Apprenticeship learning for helicopter control. Communications of the ACM , 52(7):97-105, 2009. -
- [6][6] D. Golovin and A. Krause. Adaptive submodularity: A new approach to active learning and stochastic optimization. In COLT , pages 333-345, 2010. -
- [7][7] A. Guillory and J. Bilmes. Interactive submodular set cover. In Proceedings of the International Conference on Machine Learning , 2010. -
- [8][8] M. Lopes, F. Melo, and L. Montesano. Active learning for reward estimation in inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases , pages 31-46. Springer, 2009. -
- [9][9] L. Lov´ asz. Hit-and-run mixes fast. Mathematical Programming , 86(3):443-461, 1999. -
- [10][10] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Icml , pages 663-670, 2000. -
- [12][12] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd International Conference on Machine Learning , pages 729-736. ACM, 2006. -
- [13][13] K. Regan and C. Boutilier. Regret-based reward elicitation for markov decision processes. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , pages 444-451. AUAI Press, 2009. -
- [14][14] K. Regan and C. Boutilier. Robust policy computation in reward-uncertain mdps using nondominated policies. In AAAI , 2010. -
- [15][15] K. Regan and C. Boutilier. Eliciting additive reward functions for markov decision processes. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence , volume 22, page 2159, 2011. -
- [16][16] C. A. Rothkopf and C. Dimitrakakis. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases , pages 34-48. Springer, 2011. -
- [17][17] W. D. Smart and L. P. Kaelbling. Effective reinforcement learning for mobile robots. In Robotics and Automation, 2002. Proceedings. ICRA'02. IEEE International Conference on , volume 4, pages 3404-3410. IEEE, 2002. -
- [18][18] U. Syed and R. E. Schapire. A game-theoretic 19. approach to apprenticeship learning. In Advances in neural information processing systems , pages 1449-1456, 2007. -
- [19][19] J. Von Neumann and O. Morgenstern. Theory of games and economic behavior (60th Anniversary Commemorative Edition) . Princeton university press, 2007. -
- [20][20] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI , pages 1433-1438, 2008.
Cited by
Sorted by citations
General agents contain world models
Richens, J., Abel, D., Bellot, A., & Everitt, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01622 · Cited by 1 · see citing works
- Authors
- Amin, K., & Singh, S.
- ORCID
- Singh, Satinder
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1601.06569
- OpenAlex
- W2257176436
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 1 · 2022: 1 · 2021: 2 · 2020: 3 · 2019: 1 · 2018: 1 · 2016: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- inverse reinforcement learning, non-identifiability, transition dynamics, submodular maximization, greedy algorithm, information maximization, behaviorism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
