CoChat
Sign Up
PreprintCited by 6Reinforcement Learning in RoboticsArtificial Intelligence in GamesDomain Adaptation and Few-Shot Learning

Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Open access
Dorner, Florian E.·2021·arXiv
DOI 10.48550/arxiv.2102.04881arXiv 2102.04881
Open this paper in CoChat — free accountOpen PDF

Abstract

Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms. Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efficiency. As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efficiency is an important indicator for economically relevant applications of DRL. We investigate progress in sample efficiency on Atari games and continuous control tasks by comparing the number of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications. We find exponential progress in sample efficiency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the specific task and performance level.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.04881

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 106

  1. Abbas Abdolmaleki et al. Relative entropy regularized policy iteration. arXiv preprint arXiv:1812.02256 , 2018a. · arXiv
  2. Abbas Abdolmaleki et al. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920 , 2018b. · arXiv
  3. Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning (ICML) , 2020.
  4. Michael Ahn et al. Robel: Robotics benchmarks for learning with low-cost robots. Conference on Robot Learning , 2020.
  5. Adri` a Puigdom` enech Badia et al. Agent57: Outperforming the atari human benchmark. arXiv preprint arXiv:2003.13350 , 2020. · arXiv
  6. Gabriel Barth-Maron et al. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617 , 2018. · arXiv
  7. Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich K¨ uttler, Andrew Lefrancq, Simon Green, V´ ıctor Vald´ es, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801 , 2016. · arXiv
  8. Marc G. Bellemare, Will Dabney, and R´ emi Munos. A distributional perspective on reinforcement learning. arXiv preprint arXiv:1707.06887 , 2017. · arXiv
  9. Marc G. Bellemare et al. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.
  10. Aditya Bhatt et al. Crossnorm: Normalization for off-policy td reinforcement learning. arXiv preprint arXiv:1902.05605 , 2019. · arXiv
  11. Rinu Boney, Juho Kannala, and Alexander Ilin. Regularizing model-based planning with energybased models. Conference on Robot Learning , 2020.
  12. Greg Brockman et al. Openai gym. arXiv preprint arXiv:1606.01540 , 2016. · arXiv
  13. Kamil Ciosek and Shimon Whiteson. Expected policy gradients for reinforcement learning. arXiv preprint arXiv:1801.03326 , 2018. · arXiv
  14. Ignasi Clavera et al. Model-based reinforcement learning via meta-policy optimization. arXiv preprint arXiv:1809.05214 , 2018. · arXiv
  15. Cody Coleman et al. Dawnbench: An end-to-end deep learning benchmark and competition. In SOSP MLSys Workshop . 2017. Results accessed from https://dawn.cs.stanford.edu/benchmark/ImageNet/train.html on the 12th of August 2020.
  16. Will Dabney et al. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018a.
  17. Will Dabney et al. Implicit quantile networks for distributional reinforcement learning. arXiv preprint arXiv:1806.06923 , 2018b. · arXiv
  18. Jia Deng et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , 2009.
  19. Thinh T. Doan et al. Convergence rates of accelerated markov gradient descent with applications in reinforcement learning. arXiv preprint arXiv:2002.02873 , 2020. · arXiv
  20. Yan Duan et al. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning , 2016.
  21. Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 , 2019. · arXiv
  22. Adrien Ecoffet et al. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019. · arXiv
  23. Lasse Espeholt et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018. · arXiv
  24. Lasse Espeholt et al. Seed rl: Scalable and efficient deep-rl with accelerated central inference. arXiv preprint arXiv:1910.06591 , 2019. · arXiv
  25. Johannes K. Fichte, Markus Hecher, and Stefan Szeider. A time leap challenge for sat solving. arXiv preprint arXiv:2008.02215 , 2020. · arXiv
  26. Meire Fortunato et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017. · arXiv
  27. Justin Fu et al. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020. · arXiv
  28. Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477 , 2018. · arXiv
  29. Yasuhiro Fujita and Shin-ichi Maeda. Clipped action policy gradient. arXiv preprint arXiv:1802.07564 , 2018. · arXiv
  30. Anirudh Goyal et al. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv:1901.10902 , 2019. · arXiv
  31. Jean-Bastien Grill et al. Monte-carlo tree search as regularized policy optimization. arXiv preprint arXiv:2007.12509 , 2020. · arXiv
  32. Ross Gruetzemacher and Jess Whittlestone. Defining and unpacking transformative AI. arXiv preprint arXiv:1912.00747 , 2019. · arXiv
  33. Audrunas Gruslys et al. The reactor: A sample-efficient actor-critic architecture. arXiv preprint arXiv:1704.04651 5 , 2017. · arXiv
  34. Shixiang Gu et al. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247 , 2016. · arXiv
  35. Tuomas Haarnoja et al. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv:1702.08165 , 2017. · arXiv
  36. Tuomas Haarnoja et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 2018. · arXiv
  37. Danijar Hafner et al. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning , 2019a.
  38. Danijar Hafner et al. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 , 2019b. · arXiv
  39. Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series , 2015.
  40. Nicolas Heess et al. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems . 2015.
  41. Peter Henderson et al. Deep reinforcement learning that matters. AAAI , 2018.
  42. Danny Hernandez and Tom B. Brown. Measuring the algorithmic efficiency of neural networks. arXiv preprint arXiv:2005.04305 , 2020. · arXiv
  43. Matteo Hessel et al. Rainbow: Combining improvements in deep reinforcement learning. In ThirtySecond AAAI Conference on Artificial Intelligence , 2018.
  44. Dan Horgan et al. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933 , 2018. · arXiv
  45. Stephen James et al. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-tocanonical adaptation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019.
  46. Michael Janner et al. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems . 2019.
  47. Lukasz Kaiser et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374 , 2019. · arXiv
  48. Steven Kapturowski et al. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2018.
  49. Kacper Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003.10181 , 2020. · arXiv
  50. Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 , 2020. · arXiv
  51. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems . 2012.
  52. Michael Laskin et al. Reinforcement learning with augmented data. arXiv preprint arXiv:2004.14990 , 2020. · arXiv
  53. Kimin Lee et al. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.04938 , 2020. · arXiv
  54. Sergey Levine et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020. · arXiv
  55. Xiaoyuan Liang et al. Deep reinforcement learning for traffic light control in vehicular networks. arXiv preprint arXiv:1803.11115 , 2018. · arXiv
  56. Yitao Liang et al. State of the art control of atari games using shallow reinforcement learning. arXiv preprint arXiv:1512.01563 , 2015. · arXiv
  57. Timothy P. Lillicrap et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015. · arXiv
  58. A.Rupam Mahmood et al. Benchmarking reinforcement learning algorithms on real-world robots. arXiv preprint arXiv:1809.07731 , 2018. · arXiv
  59. Fernando Martinez-Plumed et al. Accounting for the neglected dimensions of AI progress. arXiv preprint arXiv:1806.00610 , 2018. · arXiv
  60. Bogdan Mazoure et al. Leveraging exploration in off-policy algorithms via normalizing flows. Conference on Robot Learning , 2020.
  61. Volodymyr Mnih et al. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013. · arXiv
  62. Volodymyr Mnih et al. Human-level control through deep reinforcement learning. Nature , 518 (7540):529-533, 2015.
  63. Volodymyr Mnih et al. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , 2016.
  64. Ofir Nachum et al. Trust-pcl: An off-policy trust region method for continuous control. arXiv preprint arXiv:1707.01891 , 2017. · arXiv
  65. Nikolay Nikolov et al. Information-directed exploration for deep reinforcement learning. arXiv preprint arXiv:1812.07544 , 2018. · arXiv
  66. Brendan O'Donoghue et al. Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626 , 2016. · arXiv
  67. Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. arXiv preprint arXiv:2007.14535 , 2020. · arXiv
  68. OpenAI et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39(1):3-20, 2020.
  69. Xue Bin Peng et al. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , 2018.
  70. Aravind Rajeswaran et al. Towards generalization and simplicity in continuous control. In Advances in Neural Information Processing Systems . 2017.
  71. Mark Rowland, Will Dabney, and R´ emi Munos. Adaptive trade-offs in off-policy learning. In International Conference on Artificial Intelligence and Statistics , 2020.
  72. Ahmad El Sallab et al. End-to-end deep reinforcement learning for lane keeping assist. arXiv preprint arXiv:1612.04340 , 2016. · arXiv
  73. Tom Schaul et al. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015. · arXiv
  74. Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. arXiv preprint arXiv:1909.11583 , 2019. · arXiv
  75. Julian Schrittwieser et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265 , 2019. · arXiv
  76. John Schulman et al. Trust region policy optimization. In International conference on machine learning , 2015.
  77. Max Schwarzer et al. Data-efficient reinforcement learning with momentum predictive representations. arXiv preprint arXiv:2007.05929 , 2020. · arXiv
  78. Wenjie Shi, Shiji Song, and Cheng Wu. Soft policy gradient method for maximum entropy deep reinforcement learning. arXiv preprint arXiv:1909.03198 , 2019. · arXiv
  79. David Silver et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.
  80. David Silver et al. Mastering the game of go without human knowledge. Nature , 550(7676): 354-359, 2017.
  81. Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136 , 2020. · arXiv
  82. Chen Sun et al. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision , 2017.
  83. Karush Suri et al. Evolve to control: Evolution-based soft actor-critic for scalable reinforcement learning. arXiv preprint arXiv:2007.13690 , 2020. · arXiv
  84. Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction . MIT press, 2018.
  85. Yuval Tassa et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018. · arXiv
  86. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2012.
  87. Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. Is deep reinforcement learning really superhuman on atari? leveling the playing field. arXiv preprint arXiv:1908.04683 , 2019. · arXiv
  88. Hugo Touvron et al. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems . 2019.
  89. Aaron D. Tucker, Markus Anderljung, and Allan Dafoe. Social and governance implications of improved data efficiency. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , 2020.
  90. Jeroen van Baar et al. Sim-to-real transfer learning using robustified controllers in robotic tasks involving complex dynamics. In 2019 International Conference on Robotics and Automation (ICRA) , 2019.
  91. Hado P. van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Thirtieth AAAI conference on artificial intelligence , 2016.
  92. Hado P. van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems . 2019.
  93. Ashish Vaswani et al. Attention is all you need. In Advances in neural information processing systems . 2017.
  94. Oriol Vinyals et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.
  95. Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv:1906.08649 , 2019. · arXiv
  96. Ziyu Wang et al. Dueling network architectures for deep reinforcement learning. In International conference on machine learning , 2016a.
  97. Ziyu Wang et al. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224 , 2016b. · arXiv
  98. Yuhuai Wu et al. Scalable trust-region method for deep reinforcement learning using kroneckerfactored approximation. Advances in neural information processing systems , 2017.
  99. Zhongwen Xu, Hado P. van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems . 2018.
  100. Zhongwen Xu et al. Meta-gradient reinforcement learning with an objective discovered online. arXiv preprint arXiv:2007.08433 , 2020. · arXiv
  101. Brian Yang et al. Replab: A reproducible low-cost arm benchmark platform for robotic learning. arXiv preprint arXiv:1905.07447 , 2019a. · arXiv
  102. Derek Yang et al. Fully parameterized quantile function for distributional reinforcement learning. In Advances in Neural Information Processing Systems . 2019b.
  103. Denis Yarats et al. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741 , 2019. · arXiv
  104. Chuheng Zhang, Yuanqi Li, and Jian Li. Policy search by target distribution learning for continuous control. AAAI , 2020.
  105. Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. In Advances in Neural Information Processing Systems . 2019.
  106. Henry Zhu et al. The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004.12570 , 2020. · arXiv

Cited by

Sorted by citations

Authors
Dorner, F. E.
Affiliations
ETH Zurich (CH)
Corresponding author
Dorner, Florian E.
ORCID
Dorner, Florian E.
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.04881
arXiv
2102.04881
OpenAlex
W3129109047
Cited by
6
References
11
Citations by year
2024: 2 · 2023: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sample efficiency, deep reinforcement learning, progress measurement, Atari games, continuous control, benchmarking, doubling time

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review