CoChat
Sign Up
PreprintCited by 115Reinforcement Learning in RoboticsAdaptive Dynamic Programming ControlDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 8 of the 8 references it cites were found in a registry or an index.

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Open access
Andrychowicz, Marcin, Raichuk, Anton, Stańczyk, Piotr, Orsini, Manu, Girgin, Sertan, Marinier, Raphael, Hussenot, Léonard, Geist, Matthieu, Pietquin, Olivier, Michalski, Marcin, Gelly, Sylvain, Bachem, Olivier·2020·arXiv
DOI 10.48550/arxiv.2006.05990arXiv 2006.05990
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.05990

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Applying reinforcement learning algorithms to sufficiently new tasks requires substantial effort, expertise, and computational resources for tweaking hyperparameters.

    “However, applying reinforcement learning algorithms to sufficiently new tasks-such as moving from video games to robotics tasksrequires substantial effort, expertise, and computational resources for tweaking the hyperparameters of the algorithm 13 .”

    From Mastering Diverse Domains through World Models (2023) · cites this paper as [13]

    The Introduction (p. 1) says RL implementations involve many consequential design decisions and that performance can depend strongly on hyperparameter tuning. Study design (p. 2) explains why finding good configurations is challenging, and the paper reports training more than 250,000 agents. It does not establish that moving to a new task, such as from video games to robotics, specifically requires substantial expertise and computational resources for tuning. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • It is not always optimal to maximize the number of steps per second, as this may reduce sample efficiency.

    “it is not always optimal to maximize the number of steps per second, as this may reduce the sample efficiency”

    From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Andrychowicz et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [1][1] Peter J Huber. 'Robust estimation of a location parameter'. In: Breakthroughs in statistics . Springer, 1992, pp. 492-518. -
  2. [2][2] Ronald J Williams. 'Simple statistical gradient-following algorithms for connectionist reinforcement learning'. In: Machine learning 8.3-4 (1992), pp. 229-256. -
  3. [3][3] Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning . Vol. 135. MIT press Cambridge, 1998. -
  4. [5][5] Emanuel Todorov, Tom Erez, and Yuval Tassa. 'Mujoco: A physics engine for model-based control'. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE. 2012, pp. 5026-5033. -
  5. [7][7] Kaiming He et al. 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification'. In: Proceedings of the IEEE international conference on computer vision . 2015, pp. 1026-1034. -
  6. [9][9] John Schulman et al. 'High-dimensional continuous control using generalized advantage estimation'. In: arXiv preprint arXiv:1506.02438 (2015). - · arXiv
  7. [10][10] John Schulman et al. 'Trust region policy optimization'. In: International conference on machine learning . 2015, pp. 1889-1897. -
  8. [11][11] Martin Abadi et al. 'Tensorflow: A system for large-scale machine learning'. In: 12th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 16) . 2016, pp. 265-283. -
  9. [12][12] Greg Brockman et al. 'Openai gym'. In: arXiv preprint arXiv:1606.01540 (2016). - · arXiv
  10. [13][13] Timothy P Lillicrap et al. 'Continuous control with deep reinforcement learning'. In: International Conference on Learning Representations . 2016. -
  11. [14][14] Volodymyr Mnih et al. 'Asynchronous methods for deep reinforcement learning'. In: International conference on machine learning . 2016, pp. 1928-1937. -
  12. [16][16] Fabio Pardo et al. 'Time limits in reinforcement learning'. In: arXiv preprint arXiv:1712.00378 (2017). - · arXiv
  13. [18][18] Abbas Abdolmaleki et al. 'Maximum a posteriori policy optimisation'. In: arXiv preprint arXiv:1806.06920 (2018). - · arXiv
  14. [19][19] Lasse Espeholt et al. 'IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures'. In: International Conference on Machine Learning . 2018, pp. 1406-1415. -
  15. [21][21] Tuomas Haarnoja et al. 'Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor'. In: International Conference on Machine Learning . 2018, pp. 1861-1870. -
  16. [23][23] Francesco Locatello et al. 'Challenging common assumptions in the unsupervised learning of disentangled representations'. In: arXiv preprint arXiv:1811.12359 (2018). - · arXiv
  17. [24][24] Mario Lucic et al. 'Are gans created equal? a large-scale study'. In: Advances in neural information processing systems . 2018, pp. 700-709. -
  18. [25][25] MAndrychowicz OpenAI et al. 'Learning dexterous in-hand manipulation'. In: arXiv preprint arXiv:1808.00177 (2018). - · arXiv
  19. [26][26] George Tucker et al. 'The mirage of action-dependent baselines in reinforcement learning'. In: arXiv preprint arXiv:1802.10031 (2018). - · arXiv
  20. [27][27] Logan Engstrom et al. 'Implementation Matters in Deep RL: A Case Study on PPO and TRPO'. In: International Conference on Learning Representations . 2019. -
  21. [28][28] Lasse Espeholt et al. 'SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference'. In: arXiv preprint arXiv:1910.06591 (2019). - · arXiv
  22. [29][29] Michael Janner et al. 'When to trust your model: Model-based policy optimization'. In: Advances in Neural Information Processing Systems . 2019, pp. 12498-12509. -
  23. [31][31] Eric Langlois et al. 'Benchmarking model-based reinforcement learning'. In: arXiv preprint arXiv:1907.02057 (2019). - · arXiv
  24. [32][32] Ilge OpenAI et al. 'Solving Rubik's Cube with a Robot Hand'. In: arXiv preprint arXiv:1910.07113 (2019). - · arXiv
  25. [33][33] Xue Bin Peng et al. 'Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning'. In: arXiv preprint arXiv:1910.00177 (2019). - · arXiv
  26. [34][34] H Francis Song et al. 'V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control'. In: arXiv preprint arXiv:1909.12238 (2019). - · arXiv
  27. [36][36] Ilija Radosavovic et al. 'Designing Network Design Spaces'. In: arXiv preprint arXiv:2003.13678 (2020). · arXiv

Cited by

Sorted by citations

  • Mastering Diverse Domains through World Models

    Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124

    “However, applying reinforcement learning algorithms to sufficiently new tasks—such as moving from video games to robotics tasks— requires substantial effort, expertise, and computational resources for tweaking the hyperparameters of the algorithm 13 .”
    Method

  • XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX

    Nikulin, A., Kurenkov, V., Zisman, I., Agarkov, A., Sinii, V., & Kolesnikov, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1390 · Cited by 3

    “Although it is not always optimal to maximize the number of steps per second, as this may reduce the sample efficiency (Andrychowicz et al., 2020), for demonstration purposes we searched over other parameters to maximize per device utilization.”

Metadata

Authors
Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., & Bachem, O.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.05990
arXiv
2006.05990
OpenAlex
W3035700320
Cited by
115
References
8
Citations by year
2026: 2 · 2025: 11 · 2024: 26 · 2023: 26 · 2022: 15 · 2021: 28 · 2020: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • LillOA (Université de Lille (University Of Lille)) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
on-policy learning, continuous control, hyperparameter sensitivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review