The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Relative Entropy Policy Search
Open accessAbstract
Policy search is a successful approach to reinforcement learning. However, policy improvements often result in the loss of information. Hence, it has been marred by premature convergence and implausible solutions. As first suggested in the context of covariant policy gradients, many of these problems may be addressed by constraining the information loss. In this paper, we continue this path of reasoning and suggest the Relative Entropy Policy Search (REPS) method. The resulting method differs significantly from previous policy gradient approaches and yields an exact update step. It can be shown to work well on typical reinforcement learning benchmark problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
(1999) · Cited by 5,147
Barto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Baxter, L. A. (1995) · Technometrics · doi:10.1080/00401706.1995.10484354 · Cited by 2,415
(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Kakade, S. M. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 775
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Self-citation
Cited by
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
“The idea of constraining policy gradient updates was explored earlier through the use of “natural gradient” updates [46] and also by means of the Kullback-Leibler (KL) divergence [48, 88].”
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Self-citationDeep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989385 · Cited by 1,424
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114
A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients
Grondman, I., Busoniu, L., Lopes, G. A. D., & Babuska, R. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2012.2218595 · Cited by 1,042
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712
Emergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 656
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A. S., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 534
Residual Reinforcement Learning for Robot Control
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794127 · Cited by 480
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience
Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., & Fox, D. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793789 · Cited by 457
Deep spatial autoencoders for visuomotor learning
Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487173 · Cited by 430
Reinforcement Learning with Deep Energy-Based Policies
Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08165 · Cited by 392
Reinforcement learning in robotic applications: a comprehensive survey
Singh, B., Kumar, R., & Singh, V. P. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09997-9 · Cited by 392
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.00909 · Cited by 366
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., & Knoll, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3457538 · Cited by 315
Learning modular neural network policies for multi-task and multi-robot transfer
Devin, C., Gupta, A., Darrell, T., Abbeel, P., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989250 · Cited by 310
Continuous Deep Q-Learning with Model-based Acceleration
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 304
Optimal control as a graphical model inference problem
Kappen, H. J., Gómez, V., & Opper, M. (2012) · Machine Learning · doi:10.1007/s10994-012-5278-7 · Cited by 297
Off-Policy Deep Reinforcement Learning without Exploration
Fujimoto, S., Meger, D., & Precup, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02900 · Cited by 268
Learning contact-rich manipulation skills with guided policy search
Levine, S., Wagener, N., & Abbeel, P. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7138994 · Cited by 267
A Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 257
Thermodynamics as a theory of decision-making with information-processing costs
Ortega, P. A., & Braun, D. A. (2013) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2012.0683 · Cited by 255
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Nair, A., Gupta, A., Dalal, M., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09359 · Cited by 175
“The analytic solution to Equation 1 can be obtained by enforcing the KKT conditions [13, 14, 12].”
BackgroundAn information-theoretic approach to curiosity-driven reinforcement learning
Still, S., & Precup, D. (2012) · Theory in Biosciences · doi:10.1007/s12064-011-0142-z · Cited by 165
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2020) · arXiv · doi:10.48550/arxiv.2005.12729 · Cited by 146
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128
Self-citationWay Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., & Picard, R. (2019) · arXiv · doi:10.48550/arxiv.1907.00456 · Cited by 126
Data-Efficient Generalization of Robot Skills with Contextual Policy Search
Kupcsik, A., Deisenroth, M., Peters, J., & Neumann, G. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8546 · Cited by 92
Self-citationData-Efficient Domain Randomization With Bayesian Optimization
Muratore, F., Eilers, C., Gienger, M., & Peters, J. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3052391 · Cited by 77
“(REPS) [18] to update the simulator’s parameters, hence turning (1) into an RL problem.”
BackgroundSelf-citationVariance Reduction for Reinforcement Learning in Input-Driven Environments
Mao, H., Venkatakrishnan, S. B., Schwarzkopf, M., & Alizadeh, M. (2018) · arXiv · doi:10.48550/arxiv.1807.02264 · Cited by 62
“Likewise, constraining the policy gradient step [24, 41] or using guided policy search with explicit models [11, 27, 36] also improve policies in control contexts.”
BackgroundAccelerating robotic reinforcement learning via parameterized action primitives
Dalal, M., Pathak, D., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2110.15360 · Cited by 44
A Survey of Exploration Methods in Reinforcement Learning
Amin, S., Gomrokchi, M., Satija, H., van Hoof, H., & Precup, D. (2021) · arXiv · doi:10.48550/arxiv.2109.00157 · Cited by 41
Metadata
- Authors
- Peters, J., Mulling, K., & Altun, Y.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2010-07-05
- Volume
- 24
- Issue
- 1
- Pages
- 1607-1612
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W1499669280
- Cited by
- 455
- References
- 21
- FWCI
- 7.02
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.