CoChat
Sign Up
PreprintCited by 33Reinforcement Learning in RoboticsArtificial Intelligence in GamesDomain Adaptation and Few-Shot Learning

Prioritized Level Replay

Open access
Jiang, Minqi, Grefenstette, Edward, Rocktäschel, Tim·2020·arXiv
DOI 10.48550/arxiv.2010.03934arXiv 2010.03934
Open this paper in CoChat — free accountOpen PDF

Abstract

Environments with procedurally generated content serve as important benchmarks for testing systematic generalization in deep reinforcement learning. In this setting, each level is an algorithmically created environment instance with a unique configuration of its factors of variation. Training on a prespecified subset of levels allows for testing generalization to unseen levels. What can be learned from a level depends on the current policy, yet prior work defaults to uniform sampling of training levels independently of the policy. We introduce Prioritized Level Replay (PLR), a general framework for selectively sampling the next training level by prioritizing those with higher estimated learning potential when revisited in the future. We show TD-errors effectively estimate a level's future learning potential and, when used to guide the sampling procedure, induce an emergent curriculum of increasingly difficult levels. By adapting the sampling of training levels, PLR significantly improves sample efficiency and generalization on Procgen Benchmark--matching the previous state-of-the-art in test return--and readily combines with other methods. Combined with the previous leading method, PLR raises the state-of-the-art to over 76% improvement in test return relative to standard RL baselines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.03934

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Regret-based UED algorithms are effective in producing levels at the frontier of the student's capability.

    “While regret-based UED algorithms [23, 27, 28] are effective in producing levels at the frontier of the student's capability”

    From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [28]

    The Background section says prior work can improve generalization by altering levels to match an agent’s abilities, and the MiniGrid discussion says PLR’s sampled difficulty should adapt to the agent’s current abilities. But this paper does not establish that regret-based UED algorithms produce such levels; its related-work discussion distinguishes PLR’s replay of existing levels from methods that generate them. The copy provides no page numbers for these passages. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Existing UED methods mainly rely on regret to guide curriculum design.

    “Existing UED methods mainly rely on regret , a metric that measures the difference between the agent's optimal and actual performance, to guide curriculum design.”

    From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [28]

    The Background (Section 2) says prior work adapted level difficulty to agents' abilities, but does not say those methods mainly used regret. Searches for “regret” and for regret-based curriculum design found no passage defining the metric or making that broader claim; the Abstract instead describes PLR's own TD-error-based approach. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welin- der, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. Solving rubik's cube with a robot hand. CoRR, abs/1910.07113, 2019. URL http://arxiv.org/ abs/1910.07113. · arXiv
  2. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, O. P., and Zaremba, W. Hindsight experience replay. In Advances in neural information processing systems, pp. 5048–5058, 2017.
  3. Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., Parisotto, E., Dumoulin, V., Moitra, S., Hughes, E., Dunning, I., Mourad, S., Larochelle, H., Bellemare, M. G., and Bowling, M. The han- abi challenge: A new frontier for AI research. Ar- tif. Intell., 280:103216, 2020. doi: 10.1016/j.artint. 2019.103216. URL https://doi.org/10.1016/ j.artint.2019.103216. · DOI
  4. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Re- search, 47:253–279, Jun 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URL http://dx.doi.org/10. 1613/jair.3912. · DOI · Paper page
  5. Campero, A., Raileanu, R., Küttler, H., Tenenbaum, J. B., Rocktäschel, T., and Grefenstette, E. Learning with AMIGo: Adversarially Motivated Intrinsic Goals. CoRR, abs/2006.12122, 2020. URL https://arxiv.org/ abs/2006.12122. · arXiv
  6. Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. BabyAI: First steps towards grounded language learning with a human in the loop. CoRR, abs/1810.08272, 2018. URL http://arxiv.org/abs/1810.08272. · arXiv
  7. Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pp. 1282–1289. PMLR, 2019.
  8. Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging Procedural Generation to Benchmark Reinforce- ment Learning. In International Conference on Ma- chine Learning, pp. 2048–2056. PMLR, November 2020a. URL http://proceedings.mlr.press/v119/ cobbe20a.html. ISSN: 2640-3498.
  9. Cobbe, K., Hilton, J., Klimov, O., and Schulman, J. Phasic Policy Gradient. CoRR, abs/2009.04416, 2020b. URL https://arxiv.org/abs/2009.04416. · arXiv
  10. Cohn, D., Atlas, L., and Ladner, R. Improving general- ization with active learning. Machine learning, 15(2): 201–221, 1994. http://arxiv.org/abs/1910.07113 http://arxiv.org/abs/1910.07113 https://doi.org/10.1016/j.artint.2019.103216 https://doi.org/10.1016/j.artint.2019.103216 http://dx.doi.org/10.1613/jair.3912 http://dx.doi.org/10.1613/jair.3912 https://arxiv.org/abs/2006.12122 https://arxiv.org/abs/2006.12122 http://arxiv.org/abs/1810.08272 https://github.com/maximecb/gym-minigrid https://github.com/maximecb/gym-minigrid http://proceedings.mlr.press/v119/cobbe20a.html http://proceedings.mlr.press/v119/cobbe20a.html https://arxiv.org/abs/2009.04416 · DOI
  11. Dennis, M., Jaques, N., Vinitsky, E., Bayen, A. M., Russell, S., Critch, A., and Levine, S. Emergent complexity and zero-shot transfer via unsupervised environment design. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings. neurips.cc/paper/2020/hash/ 985e9a46e10005356bbaf194249f6856- Abstract.html.
  12. Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. Automated curriculum learning for neural networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Aus- tralia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1311–1320. PMLR, 2017. URL http://proceedings.mlr.press/ v70/graves17a.html.
  13. Igl, M., Ciosek, K., Li, Y., Tschiatschek, S., Zhang, C., Devlin, S., and Hofmann, K. Generalization in reinforce- ment learning with selective noise injection and infor- mation bottleneck. In Advances in Neural Information Processing Systems, pp. 13978–13990, 2019.
  14. Igl, M., Farquhar, G., Luketina, J., Boehmer, W., and Whiteson, S. The impact of non-stationarity on gen- eralisation in deep reinforcement learning. CoRR, abs/2006.05826, 2020. URL https://arxiv.org/ abs/2006.05826. · arXiv
  15. Juliani, A., Khalifa, A., Berges, V.-P., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., and Lange, D. Obstacle tower: A generalization challenge in vision, control, and planning. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug 2019. doi: 10.24963/ijcai.2019/373. URL http: //dx.doi.org/10.24963/ijcai.2019/373. · DOI · Paper page
  16. Justesen, N., Torrado, R. R., Bontrager, P., Khalifa, A., Togelius, J., and Risi, S. Procedural level generation im- proves generality of deep reinforcement learning. CoRR, abs/1806.10729, 2018. URL http://arxiv.org/ abs/1806.10729. · arXiv
  17. Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. Recurrent experience replay in distributed reinforcement learning. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum? id=r1lyTjAqYX.
  18. Khalifa, A., Bontrager, P., Earle, S., and Togelius, J. PC- GRL: procedural content generation via reinforcement learning. CoRR, abs/2001.09212, 2020. URL https: //arxiv.org/abs/2001.09212. · arXiv
  19. Kostrikov, I., Yarats, D., and Fergus, R. Image Augmenta- tion Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. CoRR, abs/2004.13649, 2020. URL https://arxiv.org/abs/2004.13649. · arXiv
  20. Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T. The nethack learning environment. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings. neurips.cc/paper/2020/hash/ 569ff987c643b4bedf504efda8f786c2- Abstract.html.
  21. Matiisen, T., Oliver, A., Cohen, T., and Schulman, J. Teacher–student curriculum learning. IEEE Transac- tions on Neural Networks and Learning Systems, 31 (9):3732–3740, Sep 2020. ISSN 2162-2388. doi: 10.1109/tnnls.2019.2934906. URL http://dx.doi. org/10.1109/TNNLS.2019.2934906. · DOI
  22. Raileanu, R. and Rocktäschel, T. RIDE: rewarding impact- driven exploration for procedurally-generated environ- ments. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: //openreview.net/forum?id=rkg-TJBFPB.
  23. Raileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., and Fergus, R. Automatic data augmentation for general- ization in reinforcement learning, 2021. URL https: //openreview.net/forum?id=9l9WD4ahJgs.
  24. Risi, S. and Togelius, J. Increasing generality in machine learning through procedural content genera- tion. Nature Machine Intelligence, 2(8):428–436, Aug 2020. ISSN 2522-5839. doi: 10.1038/s42256-020- 0208-z. URL http://dx.doi.org/10.1038/ s42256-020-0208-z. · DOI
  25. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prior- itized experience replay. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Rep- resentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs/1511.05952. · arXiv
  26. Prioritized Level Replay generalized advantage estimation. In Bengio, Y. and Le- Cun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs/1506.02438. · arXiv
  27. Sharma, S., Jha, A. K., Hegde, P., and Ravindran, B. Learning to multi-task by active sampling. In 6th In- ternational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum? id=B1nZ1weCZ.
  28. Sukhbaatar, S., Kostrikov, I., Szlam, A., and Fergus, R. Intrinsic motivation and automatic curricula via asym- metric self-play. CoRR, abs/1703.05407, 2017. URL http://arxiv.org/abs/1703.05407. · arXiv
  29. Wang, K., Kang, B., Shao, J., and Feng, J. Improving gener- alization in reinforcement learning with mixture regular- ization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020a. URL https://proceedings. neurips.cc/paper/2020/hash/ 5a751d6a0b6ef05cfe51b86e5d1458e6- Abstract.html.
  30. Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. Enhanced POET: Open-ended Reinforce- ment Learning through Unbounded Invention of Learning Challenges and their Solutions. In International Con- ference on Machine Learning, pp. 9940–9951. PMLR, November 2020b. URL http://proceedings. mlr.press/v119/wang20l.html. ISSN: 2640- 3498.
  31. Zha, D., Ma, W., Yuan, L., Hu, X., and Liu, J. Rank the episodes: A simple approach for explo- ration in procedurally-generated environments. CoRR, abs/2101.08152, 2021. URL https://arxiv.org/ abs/2101.08152. · arXiv
  32. Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonzalez, J. E., and Tian, Y. Bebold: Exploration beyond the bound- ary of explored regions. CoRR, abs/2012.08621, 2020a. URL https://arxiv.org/abs/2012.08621. · arXiv
  33. Zhang, Y., Abbeel, P., and Pinto, L. Automatic curriculum learning through value disagreement. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020b. URL https://proceedings. neurips.cc/paper/2020/hash/ 566f0ea4f6c2e947f36795c8f58ba901- Abstract.html.
  34. Zhong, V., Rocktäschel, T., and Grefenstette, E. RTFM: generalising to new environment dynamics via read- ing. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: //openreview.net/forum?id=SJgob6NKvH. http://arxiv.org/abs/1506.02438 http://arxiv.org/abs/1707.06347 http://arxiv.org/abs/1707.06347 https://openreview.net/forum?id=B1nZ1weCZ https://openreview.net/forum?id=B1nZ1weCZ http://arxiv.org/abs/1703.05407 https://proceedings.neurips.cc/paper/2020/hash/5a751d6a0b6ef05cfe51b86e5d1458e6-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/5a751d6a0b6ef05cfe51b86e5d1458e6-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/5a751d6a0b6ef05cfe51b86e5d1458e6-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/5a751d6a0b6ef05cfe51b86e5d1458e6-Abstract.html https://doi.org/10.1145/3321707.3321799 https://doi.org/10.1145/3321707.3321799 http://proceedings.mlr.press/v119/wang20l.html http://proceedings.mlr.press/v119/wang20l.html https://arxiv.org/abs/2101.08152 https://arxiv.org/abs/2101.08152 https://arxiv.org/abs/2012.08621 https://proceedings.neurips.cc/paper/2020/hash/566f0ea4f6c2e947f36795c8f58ba901-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/566f0ea4f6c2e947f36795c8f58ba901-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/566f0ea4f6c2e947f36795c8f58ba901-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/566f0ea4f6c2e947f36795c8f58ba901-Abstract.html https://openreview.net/forum?id=SJgob6NKvH https://openreview.net/forum?id=SJgob6NKvH · DOI · Paper page
  35. To make the most efficient use of our computational re- sources, we perform hyperparameter sweeps on the easy setting. This also makes our results directly comparable to most prior works benchmarked on Procgen, which have likewise focused on the easy setting. In Procgen easy, our experiments use the recommended settings of Ntrain = 200 and 25M steps of training, as well as the same ResNet pol- icy architecture and PPO hyperparameters shared across all games as in Cobbe et al. (2020a) and Raileanu et al. (2021). We find 25M steps to be sufficient for uncovering differ- ences in generalization performance among our methods and standard baselines. Moreover, under this setup, we find Procgen training runs require much less wall-clock time than training runs on the two MiniGrid environments of in- terest over an equivalent number of steps needed to uncover differences in generalization performance. Therefore we survey the empirical differences across various settings of PLR on Procgen easy rather than MiniGrid.

Cited by

Sorted by citations

Authors
Jiang, M., Grefenstette, E., & Rocktäschel, T.
Affiliations
  • Meta (United Kingdom) (GB)
  • Meta (United States) (US)
  • University College London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.03934
arXiv
2010.03934
OpenAlex
W3092271956
Cited by
33
References
2
Citations by year
2025: 5 · 2024: 7 · 2023: 5 · 2022: 6 · 2021: 9 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
procedural content generation, systematic generalization, deep reinforcement learning, curriculum learning, sample efficiency, variation factors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review