CoChat
Sign Up
PreprintCited by 804Reinforcement Learning in RoboticsAdaptive Dynamic Programming ControlMachine Learning and Algorithms

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Constrained Policy Optimization

Open access
Achiam, Joshua, Held, David, Tamar, Aviv, Abbeel, Pieter·2017·arXiv
DOI 10.48550/arxiv.1705.10528arXiv 1705.10528
Open this paper in CoChat — free accountOpen PDF

Abstract

For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1705.10528

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The MetaDrive simulator can integrate real-world driving data, enabling the simulation of realistic driving scenarios.

    “In this work, we use the MetaDrive simulator [26] as the training and evaluation environment due to its ability to integrate real-world driving data, enabling the simulation of realistic driving scenarios.”

    From Ethics-Aware Safe Reinforcement Learning for Rare-Event Risk Control in Interactive Urban Driving (2026) · cites this paper as [26]

    The Experiments section describes point-mass, ant, and humanoid agents and says the experiments were implemented in rllab. Searches for MetaDrive and for real-world driving data found no passage addressing the simulator or its claimed ability to produce realistic driving scenarios. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Altman, Eitan. Constrained Markov Decision Processes. pp. 260, 1999. ISSN 01676377. doi: 10.1016/ 0167-6377(96)00003-X. · DOI
  2. Amodei, Dario, Olah, Chris, Steinhardt, Jacob, Christiano, Paul, Schulman, John, and Mané, Dan. Concrete Prob- lems in AI Safety. arXiv, 2016. URL http://arxiv. org/abs/1606.06565.
  3. Bou Ammar, Haitham, Tutunov, Rasul, and Eaton, Eric. Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret. International Conference on Ma- chine Learning, 37:19, 2015. URL http://arxiv. org/abs/1505.0579.
  4. Chow, Yinlam, Ghavamzadeh, Mohammad, Janson, Lucas, and Pavone, Marco. Risk-Constrained Reinforcement Learning with Percentile Risk Criteria. Journal of Ma- chine Learning Research, 1(xxxx):1-49, 2015.
  5. Csiszar, I and Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems. Book, 244:452, 1981. ISSN 0895-4801. doi: 10.2307/ 2529636. URL http://www.getcited.org/ pub/102082957. · DOI
  6. Duan, Yan, Chen, Xi, Schulman, John, and Abbeel, Pieter. Benchmarking Deep Reinforcement Learning for Con- tinuous Control. The 33rd International Conference on Machine Learning (ICML 2016) (2016), 48:14, 2016. URL http://arxiv.org/abs/1604.06778. · arXiv
  7. García, Javier and Fernández, Fernando. A Comprehensive Survey on Safe Reinforcement Learning. Journal of Ma- chine Learning Research, 16:1437-1480, 2015. ISSN 15337928.
  8. Gu, Shixiang, Lillicrap, Timothy, Ghahramani, Zoubin, Turner, Richard E., and Levine, Sergey. Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic. In International Conference on Learning Repre- sentations, 2017. URL http://arxiv.org/abs/ 1611.02247.
  9. Held, David, Mccarthy, Zoe, Zhang, Michael, Shentu, Fred, and Abbeel, Pieter. Probabilistically Safe Policy Transfer. In Proceedings of the IEEE International Con- ference on Robotics and Automation (ICRA), 2017.
  10. Jiang, Nan and Li, Lihong. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. Interna- tional Conference on Machine Learning, 2015. URL http://arxiv.org/abs/1511.03722. · arXiv
  11. Kakade, Sham and Langford, John. Approximately Optimal Approximate Reinforcement Learning. Proceedings of the 19th International Conference on Machine Learning, pp. 267-274, 2002. URL http://www.cs.cmu.edu/afs/cs/Web/ People/jcl/papers/aoarl/Final.pdf.
  12. Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-End Training of Deep Visuo- motor Policies. Journal of Machine Learning Re- search, 17:1-40, 2016. ISSN 15337928. doi: 10.1007/ s13398-014-0173-7. · DOI
  13. Lillicrap, Timothy P., Hunt, Jonathan J., Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep re- inforcement learning. In International Conference on Learning Representations, 2016. ISBN 2200000006. doi: 10.1561/2200000006. · DOI
  14. Lipton, Zachary C., Gao, Jianfeng, Li, Lihong, Chen, Jianshu, and Deng, Li. Combating Deep Reinforce- ment Learning's Sisyphean Curse with Intrinsic Fear. In arXiv, 2017. ISBN 2004012439. URL http: //arxiv.org/abs/1611.01211.
  15. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei a, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-533, 2015. ISSN 0028-0836. doi: 10.1038/nature14236. URL http: //dx.doi.org/10.1038/nature14236. · DOI · Paper page
  16. Mnih, Volodymyr, Badia, Adrià Puigdomènech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P., Harley, Tim, Silver, David, and Kavukcuoglu, Koray. Asynchronous Methods for Deep Reinforcement Learning. pp. 1- 28, 2016. URL http://arxiv.org/abs/1602. 01783.
  17. Moldovan, Teodor Mihai and Abbeel, Pieter. Safe Explo- ration in Markov Decision Processes. Proceedings of the 29th International Conference on Machine Learn- ing, 2012. URL http://arxiv.org/abs/1205. 4810.
  18. Ng, Andrew Y., Harada, Daishi, and Russell, Stuart. Pol- icy invariance under reward transformations : Theory and application to reward shaping. Sixteenth Inter- national Conference on Machine Learning, 3:278-287, 1999. doi: 10.1.1.48.345.
  19. Peters, Jan and Schaal, Stefan. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21 (4):682-697, 2008. ISSN 08936080. doi: 10.1016/j. neunet.2008.02.003. · DOI · Paper page
  20. Pirotta, Matteo, Restelli, Marcello, and Calandriello, Daniele. Safe Policy Iteration. Proceedings of the 30th International Conference on Machine Learning, 28, 2013.
  21. Schulman, John, Moritz, Philipp, Jordan, Michael, and Abbeel, Pieter. Trust Region Policy Optimization. In- ternational Conference on Machine Learning, 2015.
  22. Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael, and Abbeel, Pieter. High-Dimensional Contin- uous Control Using Generalized Advantage Estimation. arXiv, 2016.
  23. Shalev-Shwartz, Shai, Shammah, Shaked, and Shashua, Amnon. Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving. arXiv, 2016. URL http: //arxiv.org/abs/1610.03295.
  24. Sutton, Richard S and Barto, Andrew G. Introduction to Reinforcement Learning. Learning, 4(1996):1-5, 1998. ISSN 10743529. doi: 10.1.1.32.7692. URL http:// dl.acm.org/citation.cfm?id=551283.
  25. Uchibe, Eiji and Doya, Kenji. Constrained reinforce- ment learning from intrinsic and extrinsic rewards. 2007 IEEE 6th International Conference on Development and Learning, ICDL, (February):163-168, 2007. doi: 10. 1109/DEVLRN.2007.4354030. · DOI

Cited by

Sorted by citations

Metadata

Authors
Achiam, J., Held, D., Tamar, A., & Abbeel, P.
Affiliations
  • University of California, Berkeley (US)
  • OpenAI (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
22-31
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1705.10528
arXiv
1705.10528
OpenAlex
W2616964725
Cited by
804
References
1
Citations by year
2026: 45 · 2025: 87 · 2024: 161 · 2023: 154 · 2022: 105 · 2021: 119 · 2020: 66 · 2019: 40 · 2018: 24 · 2017: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Funders
National Science Foundation (0424422)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
constrained reinforcement learning, constrained policy optimization, policy search, safety constraints, high-dimensional control, neural network policy, policy divergence, robot locomotion, human-robot interaction safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review