Identifiability in inverse reinforcement learning
Open accessAbstract
Inverse reinforcement learning attempts to reconstruct the reward function in a Markov decision problem, using observations of agent actions. As already observed in Russell [1998] the problem is ill-posed, and the reward function is not identifiable, even under the presence of perfect information about optimal behavior. We provide a resolution to this non-identifiability for problems with entropy regularization. For a given environment, we fully characterize the reward functions leading to a given policy and demonstrate that, given demonstrations of actions for the same reward under two distinct discount factors, or under sufficiently different environments, the unobserved reward can be recovered up to a constant. We also give general necessary and sufficient conditions for reconstruction of time-homogeneous rewards on finite horizons, and for action-independent rewards, generalizing recent results of Kim et al. [2021] and Fu et al. [2018].
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most data sources are insufficient for identifying the underlying reward function uniquely, even in the limit of infinite data, and this irreducible ambiguity may be problematic.
“In particular, most data sources are insufficient for identifying the underlying reward function uniquely, even in the limit of infinite data, and this irreducible ambiguity may be problematic (Ng & Russell, 2000;Dvijotham & Todorov, 2010;Cao et al., 2021;Kim et al., 2021;Skalse et al., 2023;Schlaginhaufen & Kamgarpour, 2023).”
From Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems (2024) · cites this paper as Cao et al., 2021
The Introduction (p. 2) says that even perfect knowledge of an optimal policy does not fully identify the reward, and Section 7.1 (p. 24) shows that this is not statistical error. The Introduction (p. 1) explains why identifying the specific reward matters for predictions in changed environments. The paper establishes this ambiguity for particular IRL observations, not the broader claim that most data sources are insufficient; it also describes conditions under which rewards can be identified. (LLM, conf=0.84)
Experimental — accuracy not yet measuredReward is under-determined by behavior without intervention.
“In other words, reward is under-determined by behavior (Cao et al., 2021) without intervention (Amin et al., 2017).”
From Agency Is Frame-Dependent (2025) · cites this paper as Cao et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 25
- Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.
- Kareem Amin and Satinder Singh. Towards resolving unidentifiability in inverse reinforcement learning. arXiv preprint arXiv:1601.06569 , 2016. · arXiv
- Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. Advances in Neural Information Processing Systems , 30:1815-1824, 2017.
- Sreejith Balakrishnan, Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Harold Soh. Efficient exploration of reward functions in inverse reinforcement learning via bayesian optimization. Advances in Neural Information Processing Systems , 33, 2020.
- Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 182-189. JMLR Workshop and Conference Proceedings, 2011.
- Paul Dupuis and Richard S Ellis. A weak convergence approach to the theory of large deviations , volume 902. John Wiley & Sons, 2011.
- Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solvable mdps. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML'10, page 335-342, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.
- Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 , 2016a. · arXiv
- Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning , pages 49-58, 2016b.
- Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations , 2018.
- Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning , pages 1352-1361. PMLR, 2017.
- R. E. Kalman. When Is a Linear Control System Optimal? Journal of Basic Engineering , 86(1): 51-60, 03 1964.
- Ralph L. Keeney and Howard Raiffa. Decisions with Multiple Objectives: Preferences and Value Trade-Offs . Wiley, 1976.
- Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5496-5505. PMLR, 18-24 Jul 2021. URL http://proceedings.mlr.press/ v139/kim21c.html .
- Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909 , 2018. · arXiv
- Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems , 24:19-27, 2011.
- Robert E. Lucas. Econometric policy evaluation: A critique. Carnegie-Rochester Conference Series on Public Policy , 1:19-46, 1976.
- Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of Seventeenth International Conference on Machine Learning . Citeseer, 2000.
- Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In In Proceedings of the Sixteenth International Conference on Machine Learning , pages 278-287. Morgan Kaufmann, 1999.
- Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.
- Stuart Russell. Learning agents for uncertain environments. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory , pages 101-103, 1998.
- Thomas J Sargent. Estimation of dynamic labor demand schedules under rational expectations. Journal of Political Economy , 86(6):1009-1044, 1978.
- E. Seneta. Non-negative Matrices and Markov chains . Springer, revised printing edition, 2006.
- Brian D Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy . PhD thesis, Carnegie Mellon University, 2010.
- Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
Cited by
Sorted by citations
Beyond Preferences in AI Alignment
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 85 · see citing works
Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., Omohundro, S., Szegedy, C., Goldhaber, B., Ammann, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06624 · Cited by 12 · see citing works
Ruiz-Serra, J., & Harré, M. (2023) · Algorithms · doi:10.3390/a16020068 · Cited by 9 · see citing works
Misspecification in Inverse Reinforcement Learning
Skalse, J., & Abate, A. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26766 · Cited by 7 · see citing works
Differentiable Moving Horizon Estimation for Vehicle Kinematics via Learning Covariance Matrices
Jeong, S., & Choi, J. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3343438 · Cited by 3 · see citing works
Adamczyk, J., Arriojas, A., Tiomkin, S., & Kulkarni, R. V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25817 · Cited by 3 · see citing works
Inference of Utilities and Time Preference in Sequential Decision-MakingPreprint
Cao, H., Wu, Z., & Xu, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4840776 · Cited by 2 · see citing works
Self-citationAbel, D., Barreto, A., Bowling, M., Dabney, W., Dong, S., Hansen, S., Harutyunyan, A., Khetarpal, K., Lyle, C., Pascanu, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04403
“In other words, reward is under-determined by behavior (Cao et al., 2021) without (Amin et al., 2017).”
AI in Corporate Governance: Can Machines Recover Corporate Purpose?Preprint
Nikolov, B., Schuerhoff, N., & Wagner, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5166191
- Authors
- Cao, H., Cohen, S. N., & Szpruch, L.
- Affiliations
- The Alan Turing Institute (GB)
- University of Oxford (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2106.03498
- OpenAlex
- W3212329644
- Cited by
- 7
- References
- 0
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Funders
- University of Oxford (EP/N510129/1)
- Engineering and Physical Sciences Research Council (N510129, EP/N510129/, EP/N510129/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- inverse reinforcement learning, Markov decision process, entropy regularization, discount factor, policy reconstruction, ill-posed problems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
