RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
Open accessAbstract
Offline methods for reinforcement learning have a potential to help bridge the gap between reinforcement learning research and real-world applications. They make it possible to learn policies from offline datasets, thus overcoming concerns associated with online data collection in the real-world, including cost, safety, or ethical concerns. In this paper, we propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games (e.g., Atari benchmark) and simulated motor control problems (e.g., DM Control Suite). The datasets include domains that are partially or fully observable, use continuous or discrete actions, and have stochastic vs. deterministic dynamics. We propose detailed evaluation protocols for each domain in RL Unplugged and provide an extensive analysis of supervised learning and offline RL methods using these protocols. We will release data for all our tasks and open-source all algorithms presented in this paper. We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget, thus making RL research both more systematic and more accessible across the community. Moving forward, we view RL Unplugged as a living benchmark suite that will evolve and grow with datasets contributed by the research community and ourselves. Our project page is available on https://git.io/JJUhd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Prudencio, R. F., Máximo, M. R. O. A., & Colombini, E. L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250269 · Cited by 369 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
Florence, P., Lynch, C., Zeng, A., Ramirez, O., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., & Tompson, J. (2021) · arXiv · doi:10.48550/arxiv.2109.00137 · Cited by 131 · see citing works
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., & Russell, S. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3185139 · Cited by 68 · see citing works
Is Pessimism Provably Efficient for Offline RL?Preprint
Jin, Y., Yang, Z.-R., & Wang, Z.-R. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2012.15085 · Cited by 454 · see citing works
Self-citationA Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63 · see citing works
Offline Reinforcement Learning from Images with Latent Space ModelsPreprint
Rafailov, R., Yu, T., Rajeswaran, A., & Finn, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11547 · Cited by 43 · see citing works
Model-Based Offline PlanningPreprint
Argenson, A., & Dulac-Arnold, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05556 · Cited by 44 · see citing works
Self-citationA survey of demonstration learning
Correia, A., & Alexandre, L. A. (2024) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2024.104812 · Cited by 34 · see citing works
Online and Offline Reinforcement Learning by Planning with a Learned ModelPreprint
Schrittwieser, J., Hubert, T., Mandhane, A., Barekatain, M., Antonoglou, I., & Silver, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06294 · Cited by 33 · see citing works
Conservative Data Sharing for Multi-Task Offline Reinforcement LearningPreprint
Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08128 · Cited by 18 · see citing works
Offline Evaluation for Reinforcement Learning-Based Recommendation
Deffayet, R., Thonet, T., Renders, J.-M., & de Rijke, M. (2022) · ACM SIGIR Forum · doi:10.1145/3582900.3582905 · Cited by 17 · see citing works
An Offline-Transfer-Online Framework for Cloud-Edge Collaborative Distributed Reinforcement Learning
Zeng, T., Zhang, X., Duan, J., Yu, C., Wu, C., & Chen, X. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3360438 · Cited by 17 · see citing works
Scalable Reinforcement Learning Framework for Traffic Signal Control Under Communication Delays
Pang, A., Wang, M., Chen, Y., Pun, M.-O., & Lepech, M. D. (2024) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2024.3368693 · Cited by 12 · see citing works
Value Iteration in Continuous Actions, States and TimePreprint
Lutter, M., Mannor, S., Peters, J., Fox, D., & Garg, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04682 · Cited by 12 · see citing works
Nayebi, A., Kong, N. C. L., Zhuang, C., Gardner, J. L., Norcia, A. M., & Yamins, D. L. K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.06.16.448730 · Cited by 11 · see citing works
Regularized Behavior Value EstimationPreprint
Gülçehre, Ç., Colmenarejo, S. G., Wang, Z., Sygnowski, J., Paine, T., Żołna, K., Chen, Y., Hoffman, M. W., Pascanu, R., & de Freitas, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.09575 · Cited by 11 · see citing works
Self-citationBehavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLPreprint
Cang, C., Rajeswaran, A., Abbeel, P., & Laskin, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09119 · Cited by 9 · see citing works
Learning Control Policies for Variable Objectives from Offline Data
Weber, M., Swazinna, P., Hein, D. M., Udluft, S., & Sterzing, V. (2023) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci52147.2023.10371978 · Cited by 8 · see citing works
Provable Representation Learning for Imitation with Contrastive Fourier FeaturesPreprint
Nachum, O., & Yang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12272 · Cited by 8 · see citing works
Self-citationRethinking exploration–exploitation trade-off in reinforcement learning via cognitive consistency
Wang, D. Z., Wei, W., Li, L., Wang, X., & Liang, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107342 · Cited by 7 · see citing works
On Multi-objective Policy Optimization as a Tool for Reinforcement Learning.Preprint
Abdolmaleki, A., Huang, S. H., Vezzani, G., Shahriari, B., Springenberg, J. T., Mishra, S., Tb, D., Byravan, A., Bousmalis, K., György, A., et al. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
Self-citationAdvances in Preference-based Reinforcement Learning: A Review
Abdelkareem, Y., Shehata, S., & Karray, F. O. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945333 · Cited by 5 · see citing works
Continuous-Time Fitted Value Iteration for Robust Policies
Lutter, M., Belousov, B., Mannor, S., Fox, D., Garg, A., & Peters, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3215769 · Cited by 5 · see citing works
Sun, C., Qian, H., & Miao, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29437 · Cited by 4 · see citing works
A Study of Model Based and Model Free Offline Reinforcement Learning
Shukla, I., Dozier, H. R., & Henslee, A. C. (2022) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci58124.2022.00061 · Cited by 4 · see citing works
On Instrumental Variable Regression for Deep Offline Policy EvaluationPreprint
Chen, Y., Xu, L., Gülçehre, Ç., Paine, T. L., Gretton, A., de Freitas, N., & Arnaud Doucet, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10148 · Cited by 4 · see citing works
Self-citationREALab: An Embedded Perspective on TamperingPreprint
Kumar, R., Uesato, J., Ngo, R., Everitt, T., Krakovna, V., & Legg, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.08820 · Cited by 3 · see citing works
Active Offline Policy SelectionPreprint
Konyushkova, K., Chen, Y., Paine, T. L., Gülçehre, Ç., Păduraru, C., Mankowitz, D. J., Denil, M., & de Freitas, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10251 · Cited by 3 · see citing works
Self-citationScenic4RL: Programmatic Modeling and Generation of Reinforcement Learning EnvironmentsPreprint
Azad, A. S., Kim, E., Wu, Q., Lee, K., Stoica, I. G., Abbeel, P., & Seshia, S. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10365 · Cited by 3 · see citing works
Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget MattersPreprint
Kurenkov, V., & Kolesnikov, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04156 · Cited by 3 · see citing works
Real-Data-Driven Offline Reinforcement Learning for Autonomous Vehicle Speed Decision Making
Hao, J., Xu, S., Chen, X., Fu, S., & Yang, D. (2024) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc62350.2024.10588065 · Cited by 2 · see citing works
Reinforcement Learning for decision support in defense and security: A systematic review
Schadd, M. P. D., Berman, D. S., Chen, C. M., Cohen, M., Dorsch, J., GEGOV, A. E., Moll, M., Rose, O., Rösner, A., Saastamoinen, K., et al. (2026) · Annals of Operations Research · doi:10.1007/s10479-026-07205-x · Cited by 4 · see citing works
Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
de Lima, L. M., & Krohling, R. A. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534159 · Cited by 1 · see citing works
Hyperparameter Tuning in Offline Reinforcement Learning
Tittaferrante, A., & Yassine, A. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00101 · Cited by 1 · see citing works
Model Based and Model Free Offline Reinforcement Learning Combat Simulation Scenario
Shukla, I., Henslee, A. C., Dozier, H. R., Hansen, B., Jabour, J. E., & Dettwiller, I. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00100 · Cited by 1 · see citing works
DCUR: Data Curriculum for Teaching via Samples with Reinforcement LearningPreprint
Seita, D., Gopal, A., Mandi, Z., & Canny, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07380
Offline Imitation Learning Using Reward-free Exploratory Data
Wang, H., Feng, D., Ding, B., & Li, W. (2022) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3579654.3579753
Mixed Time-Frame Training for Reinforcement Learning
Senthilnathan, G. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00042
How to Select the Appropriate One from the Trained Models for Model-Based OPE
Li, C., Wang, Y., Ma, Z.-M., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-99-9119-8_26
Automated Negotiation in Supply Chains A Generalist Environment for RL/MARL Research
Mohammad, Y., Nakadai, S., & Greenwald, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77367-9_2
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
Sinha, S., Mandlekar, A., & Garg, A. (2021) · arXiv · doi:10.48550/arxiv.2103.06326 · Cited by 5 · see citing works
- Authors
- Gulcehre, C., Wang, Z., Novikov, A., Paine, T. L., Colmenarejo, S. G., Zolna, K., Agarwal, R., Merel, J., Mankowitz, D., Paduraru, C., Dulac-Arnold, G., Li, J., Norouzi, M., Hoffman, M., Nachum, O., Tucker, G., Heess, N., & de Freitas, N.
- Affiliations
- Wuhan University (CN)
- Russian Academy of Sciences (RU)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 33
- Pages
- 7248-7259
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.13888
- OpenAlex
- W3101009114
- Cited by
- 42
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 7 · 2023: 4 · 2022: 8 · 2021: 17 · 2020: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- offline reinforcement learning, benchmark suite, motor control, supervised learning comparison, evaluation protocol, reproducibility, open-source dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
