CoChat
Sign Up
PreprintCited by 13Reinforcement Learning in RoboticsArtificial Intelligence in GamesAI-based Problem Solving and Planning

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

POPGym: Benchmarking Partially Observable Reinforcement Learning

Open access
Morad, Steven, Kortvelesy, Ryan, Bettini, Matteo, Liwicki, Stephan, Prorok, Amanda·2023·arXiv
DOI 10.48550/arxiv.2303.01859arXiv 2303.01859
Open this paper in CoChat — free accountOpen PDF

Abstract

Real world applications of Reinforcement Learning (RL) are often partially observable, thus requiring memory. Despite this, partial observability is still largely ignored by contemporary RL benchmarks and libraries. We introduce Partially Observable Process Gym (POPGym), a two-part library containing (1) a diverse collection of 15 partially observable environments, each with multiple difficulties and (2) implementations of 13 memory model baselines -- the most in a single RL library. Existing partially observable benchmarks tend to fixate on 3D visual navigation, which is computationally expensive and only one type of POMDP. In contrast, POPGym environments are diverse, produce smaller observations, use less memory, and often converge within two hours of training on a consumer-grade GPU. We implement our high-level memory API and memory baselines on top of the popular RLlib framework, providing plug-and-play compatibility with various training algorithms, exploration strategies, and distributed training paradigms. Using POPGym, we execute the largest comparison across RL memory models to date. POPGym is available at https://github.com/proroklab/popgym.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.01859

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In POPGym's Battleship task, the memory length is upper bounded by the area of the board, i.e. 12^2 = 144, and credit assignment is short-term.

    “This requires a memory length upper bounded by the area of the board, i.e. 12 2 = 144, and short-term credit assignment.”

    From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Morad et al., 2023

  • In POPGym's Repeat First task, although the reward memory length is T, the optimal policy memory length can be as low as 2.

    “Although this task indeed has a reward memory length of T , the optimal policy memory length can be as low as 2 simply by recalling the previous optimal action.”

    From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Morad et al., 2023

  • In POPGym's Autoencode task, the credit assignment length is 1, and the optimal policy memory length is at most twice the number of cards, which is 6 decks (311 steps).

    “Here, the credit assignment length is 1, and the optimal policy memory length is at most twice the number of cards, which is 6 decks (311 steps).”

    From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Morad et al., 2023

  • In POPGym's Repeat First task, the credit assignment length is 1.

    “Thus, the credit assignment length is 1.”

    From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Morad et al., 2023

  • POPGym comprises 15 tasks.

    “This benchmark comprises 15 tasks from which we examine several representative ones:”

    From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Morad et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. · arXiv
  2. Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5):834-846, 1983. Publisher: IEEE.
  3. Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. DeepMind Lab. Technical Report arXiv:1612.03801, arXiv, December 2016. URL http://arxiv.org/abs/1612.03801. arXiv:1612.03801 [cs] type: article. · arXiv
  4. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253- 279, June 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URL https://www.jair.org/ index.php/jair/article/view/10819. · DOI · Paper page
  5. Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning. The 16th International Symposium on Distributed Autonomous Robotic Systems, 2022. Publisher: Springer.
  6. Manuel Blum and Dexter Kozen. On the power of the compass (or, why mazes are easier to search than graphs). In 19th Annual Symposium on Foundations of Computer Science (sfcs 1978), pp. 132-142, October 1978. doi: 10.1109/SFCS.1978.30. ISSN: 0272-5428. · DOI
  7. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, 2016. eprint: arXiv:1606.01540. · arXiv
  8. Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. February 2022. URL https://openreview.net/forum?id=H1lJJnR5Ym.
  9. Anthony R Cassandra. A survey of POMDP applications. In Working notes of AAAI 1998 fall symposium on planning with partially observable Markov decision processes, volume 1724, 1998.
  10. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. · arXiv
  11. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging Procedural Generation to Benchmark Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 2048-2056. PMLR, November 2020. URL https://proceedings. mlr.press/v119/cobbe20a.html. ISSN: 2640-3498.
  12. Kenji Doya. Temporal Difference Learning in Continuous Time and Space. In D. Touretzky, M. C. Mozer, and M. Hasselmo (eds.), Advances in Neural Information Processing Systems, volume 8. MIT Press, 1995. URL https://proceedings.neurips.cc/paper/1995/file/ 1e1d184167ca7676cf665225e236a3d2-Paper.pdf.
  13. Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990. Publisher: Wiley Online Library.
  14. Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability. In Advances in Neural Information Processing Systems, volume 34, pp. 25502-25515.
  15. Curran As- sociates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ d5ff135377d39f1de7372c95c74dd962-Abstract.html.
  16. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, and others. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471-476, 2016. Publisher: Nature Publishing Group.
  17. Albert Gu, Karan Goel, and Christopher Re. Efficiently Modeling Long Sequences with Structured State Spaces. March 2022. URL https://openreview.net/forum?id= uYLFoz1vlAC.
  18. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Confer- ence on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18, pp. 3207-3214, New Orleans, Louisiana, USA, February 2018. AAAI Press. ISBN 978-1-57735-800-8.
  19. Chi Jin, Sham Kakade, Akshay Krishnamurthy, and Qinghua Liu. Sample-Efficient Reinforcement Learning of Undercomplete POMDPs. In Advances in Neural In- formation Processing Systems, volume 33, pp. 18530-18539.
  20. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ d783823cc6284b929c2cd8df2167d212-Abstract.html.
  21. Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? IEEE Robotics and Automation Letters, 5(4):6670- 6677, October 2020. ISSN 2377-3766. doi: 10.1109/LRA.2020.3013848. Conference Name: IEEE Robotics and Automation Letters. · DOI · Paper page
  22. Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998. Publisher: Elsevier.
  23. Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNING. pp. 19, 2019.
  24. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Proceedings of the 37th International Conference on Machine Learning, pp. 5156-5165. PMLR, November 2020. URL https://proceedings.mlr.press/v119/katharopoulos20a.html. ISSN: 2640-3498.
  25. Micha\l Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. ViZ- Doom: A Doom-based AI Research Platform for Visual Reinforcement Learning. In IEEE Con- ference on Computational Intelligence and Games, pp. 341-348, Santorini, Greece, September 2016. IEEE. URL http://arxiv.org/abs/1605.02097. · arXiv
  26. Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor. RL for Latent MDPs: Regret Guarantees and a Lower Bound. In Advances in Neu- ral Information Processing Systems, volume 34, pp. 24523-24534.
  27. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ cd755a6c6b699f3262bcc2aa46ab507e-Abstract.html.
  28. Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
  29. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, November 2020. URL http://arxiv. org/abs/2005.01643. arXiv:2005.01643 [cs, stat]. · arXiv
  30. Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5457-5466, Salt Lake City, UT, June 2018. IEEE. ISBN 978-1-5386-6420-9. doi: 10.1109/CVPR.2018.00572. URL https://ieeexplore. ieee.org/document/8578670/. · DOI · Paper page
  31. Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp. 3053-3062. PMLR, 2018.
  32. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, and others. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015. Publisher: Nature Publishing Group.
  33. Steven Morad, Stephan Liwicki, Ryan Kortvelesy, Roberto Mecca, and Amanda Prorok. Modeling Partially Observable Systems using Graph-Based Memory and Topological Priors. In Proceedings of The 4th Annual Learning for Dynamics and Control Conference, pp. 59-73. PMLR, May 2022. URL https://proceedings.mlr.press/v168/morad22a.html. ISSN: 2640-3498.
  34. Steven D. Morad, Roberto Mecca, Rudra P. K. Poudel, Stephan Liwicki, and Roberto Cipolla. Em- bodied Visual Navigation With Automatic Curriculum Learning in Real Environments. IEEE Robotics and Automation Letters, 6(2):683-690, April 2021. ISSN 2377-3766. doi: 10.1109/ LRA.2020.3048662. Conference Name: IEEE Robotics and Automation Letters. · DOI
  35. Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. In Proceedings of the 39th International Conference on Machine Learning, pp. 16691-16723. PMLR, June 2022. URL https://proceedings. mlr.press/v162/ni22a.html. ISSN: 2640-3498.
  36. Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour Suite for Reinforcement Learning. In Interna- tional Conference on Learning Representations, 2020. URL https://openreview.net/ forum?id=rygf-kSYwH.
  37. Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing Transformers for Reinforcement Learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7487-7498. PMLR, July 2020. URL https://proceedings.mlr.press/v119/parisotto20a.html.
  38. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/ 20-1364.html.
  39. Dhruv Ramani. A Short Survey On Memory Based Reinforcement Learning. Technical Re- port arXiv:1904.06736, arXiv, April 2019. URL http://arxiv.org/abs/1904.06736. arXiv:1904.06736 [cs] type: article. · arXiv
  40. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers Are Secretly Fast Weight Programmers. In Proceedings of the 38th International Conference on Machine Learning, pp. 9355-9366. PMLR, July 2021. URL https://proceedings.mlr.press/v139/ schlag21a.html. ISSN: 2640-3498.
  41. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  42. Vatsal Sharan, Sham M Kakade, Percy S Liang, and Gregory Valiant. Learning Overcomplete HMMs. In Advances in Neural Information Processing Systems, volume 30.
  43. Curran Asso- ciates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 6aca97005c68f1206823815f66102863-Abstract.html.
  44. Aleksandrs Slivkins and others. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning, 12(1-2):1-286, 2019. Publisher: Now Publishers, Inc.
  45. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
  47. Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre Memory Units: Continuous-Time Rep- resentation in Recurrent Neural Networks. In Advances in Neural Information Processing Sys- tems, volume 32.
  48. Curran Associates, Inc., 2019. URL https://papers.nips.cc/paper/ 2019/hash/952285b9b7e7a1be5aa7849f32ffff05-Abstract.html.
  49. Zhihan Yang and Hai Nguyen. Recurrent Off-policy Baselines for Memory-based Continuous Con- trol. Technical Report arXiv:2110.12628, arXiv, October 2021. URL http://arxiv.org/ abs/2110.12628. arXiv:2110.12628 [cs] type: article. · arXiv
  50. Kaiyu Zheng and Stefanie Tellex. pomdp py: A Framework to Build and Solve POMDP Problems. In ICAPS 2020 Workshop on Planning and Robotics (PlanRob), 2020. URL https://icaps20subpages.icaps-conference.org/wp-content/ uploads/2020/10/14-PlanRob_2020_paper_3.pdf.

Cited by

Sorted by citations

Metadata

Authors
Morad, S., Kortvelesy, R., Bettini, M., Liwicki, S., & Prorok, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2303.01859
arXiv
2303.01859
OpenAlex
W4323362941
Cited by
13
References
3
Citations by year
2026: 3 · 2025: 4 · 2024: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
partially observable environments, partially observable Markov decision processes, reinforcement learning benchmarks, memory models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review