Benchmarking Deep Reinforcement Learning for Continuous Control
Open accessAbstract
Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuous control due to the lack of a commonly adopted benchmark. In this work, we present a benchmark suite of continuous control tasks, including classic tasks like cart-pole swing-up, tasks with very high state and action dimensionality such as 3D humanoid locomotion, tasks with partial observations, and tasks with hierarchical structure. We report novel findings based on the systematic evaluation of a range of implemented reinforcement learning algorithms. Both the benchmark and reference implementations are released at https://github.com/rllab/rllab in order to facilitate experimental reproducibility and to encourage adoption by other researchers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Van Baar et al. [56] shows benefits of using DR and A3C algorithm [86] (on-policy) for learning the controller, with respect to not using DR.
“Van Baar et al. [56] shows the bene ts of using DR and Asynchronous Actor Critic Agent (A3C) algorithm [86] (on-policy) for learning the controller, with respect to not using DR.”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [86]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Completely Derandomized Self-Adaptation in Evolution Strategies
Hansen, N., & Ostermeier, A. (2001) · Evolutionary Computation · doi:10.1162/106365601750190398 · Cited by 4,397 · see citing works
Purcell, E. M. (1977) · American Journal of Physics · doi:10.1119/1.10903 · Cited by 4,360 · see citing works
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · see citing works
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 3× · see citing works
“Examples include MNIST (LeCun et al., 1998), Caltech101 (Fei-Fei et al., 2006), CIFAR (Krizhevsky & Hinton, 2009), ImageNet (Deng et al., 2009), PASCAL VOC (Everingham et al., 2010), BSDS500 (Martin et al., 2001), SWITCHBOARD (Godfrey et al., 1992), TIMIT (Garofolo et al., 1993), Aurora (Hirsch & Pearce, 2000), and Vo…”
BackgroundMemory-based control with recurrent neural networks
Heess, N., Hunt, J. J., Lillicrap, T. P., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1512.04455 · Cited by 211 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Self-citationDeep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · see citing works
Self-citationLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847 · see citing works
Self-citationDeep Learning for Intelligent Wireless Networks: A Comprehensive Survey
Mao, Q., Hu, F., & Hao, Q. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2846401 · Cited by 789 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735 · see citing works
Rathore, M. M., Shah, S. A., Shukla, D., Bentafat, E., & Bakiras, S. (2021) · IEEE Access · doi:10.1109/access.2021.3060863 · Cited by 616 · see citing works
Cheng, R., Orosz, G., Murray, R. M., & Burdick, J. W. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013387 · Cited by 611 · see citing works
“This method has been successfully applied to continuous control tasks (Duan et al. 2016; Lillicrap et al. 2015) where controllers have learned to stabilize complex robots (after many policy iterations).”
MethodTask Agnostic Meta-Learning for Few-Shot Learning
Jamal, M. A., & Qi, G.-J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01199 · Cited by 519 · see citing works
SAPIEN: A SimulAted Part-Based Interactive ENvironment
Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01111 · Cited by 406 · see citing works
End-to-End Race Driving with Deep Reinforcement Learning
Jaritz, M., de Charette, R., Toromanoff, M., Perot, E., & Nashashibi, F. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460934 · Cited by 210 · see citing works
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“[52] evaluate their approach on various RL tasks: rllab [111], OpenAIGym, and DartEnv [82].”
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
Islam, R., Henderson, P., Gomrokchi, M., & Precup, D. (2017) · arXiv · doi:10.48550/arxiv.1708.04133 · Cited by 177 · see citing works
Thananjeyan, B., Garg, A., Krishnan, S., Chen, C., Miller, L., & Goldberg, K. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989275 · Cited by 144 · see citing works
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · see citing works
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
Learning to dress: Synthesizing human dressing motion via deep reinforcement learning
Clegg, A., Yu, W., Tan, J., Liu, C. K., & Turk, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275048 · Cited by 89 · see citing works
Yuan, W., Hang, K., Kragic, D., Wang, M. Y., & Stork, J. A. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.06.007 · Cited by 54 · see citing works
Wang, Z., Li, H.-X., & Chen, C. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2927320 · Cited by 39 · see citing works
Zhou, Z., Oguz, O. S., Leibold, M., & Buss, M. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.2992981 · Cited by 18 · see citing works
Deep reinforcement learning for task-based feature learning in prosthetic vision
White, J., Kameneva, T., & McCarthy, C. (2019) · 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856541 · Cited by 14 · see citing works
Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
Muller-Brockhausen, M., Preuss, M., & Plaat, A. (2021) · 2021 IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619000 · Cited by 11 · see citing works
Employing Intelligent Aerial Data Aggregators for the Internet of Things: Challenges and Solutions
Li, K., Ni, W., Noor, A., & Guizani, M. (2022) · IEEE Internet of Things Magazine · doi:10.1109/iotm.001.2100161 · Cited by 11 · see citing works
Exploring the Impact of Virtualization on the Usability of Deep Learning Applications
Samani, D. G., & Salehi, M. A. (2022) · 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid) · doi:10.1109/ccgrid54584.2022.00054 · Cited by 7 · see citing works
Dizon-Paradis, O., Wormald, S., Capecci, D., Bhandarkar, A., & Woodard, D. (2023) · doi:10.36227/techrxiv.23739099 · Cited by 1 · see citing works
- Authors
- Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- Ghent University (BE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1604.06778
- OpenAlex
- W2342662072
- Cited by
- 1,114
- References
- 8
- Citations by year
- 2025: 45 · 2024: 96 · 2023: 100 · 2022: 123 · 2021: 154 · 2020: 185 · 2019: 162 · 2018: 153 · 2017: 83 · 2016: 13
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Fonds Wetenschappelijk Onderzoek
- Vlaamse regering
- Defense Advanced Research Projects Agency
- Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- continuous control, deep reinforcement learning, benchmark suite, partial observability, hierarchical control, algorithm comparison, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
