D4RL: Datasets for Deep Data-Driven Reinforcement Learning
Open accessThe DOI matches its DataCite record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples. This serves as a common starting point for the community to identify shortcomings in existing offline RL methods and a collaborative route for progress in this emerging area.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 10×
“For baseline algorithms, we use offline soft actor-critic (SAC) [Haarnoja et al., 2018], an off-policy actorcritic algorithm adapted to train exclusively from offline data, bootstrapping error accumulation reduction (BEAR) [Kumar et al., 2019], an actor-critic algorithm that uses a support constraint for the policy, a…”
MethodInfluentialSelf-citationMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Congested traffic states in empirical observations and microscopic simulations
Treiber, M., Hennecke, A., & Helbing, D. (2000) · Physical Review E · doi:10.1103/physreve.62.1805 · Cited by 4,762 · cited 1×
“Thus, we include “human” data from agents controlled by the intelligent driver model (IDM) [34], a hand-designed model of human driving behavior.”
MethodAddressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · UvA-DARE (University of Amsterdam) · Cited by 4,017
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 2×
“wn as the discounted returns. In episodic RL, the algorithm is given access to the MDP via trajectory samples for arbitrary ˇ of the algorithm’s choosing. Off-policy methods may use experience replay [22] to store these trajectories in a replay buffer Dof transitions (s t;a t;s t+1;r t), and use an off-policy algorith…”
MethodDeep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989385 · Cited by 1,424
Self-citationOn the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution ShiftPreprint
Agarwal, A., Kakade, S. M., Lee, J. D., & Mahajan, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00261 · Cited by 907
Cited by
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
“However, even this formulation does not fully evaluate capabilities of offline reinforcement learning methods, and the statistics of the training data have a considerable effect on the difficult of offline RL (Fu et al., 2020), including how concentrated the data distribution is around a specific set of trajectories,…”
BackgroundInfluentialSelf-citationConservative Q-Learning for Offline Reinforcement Learning
Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04779 · Cited by 543
Self-citationA Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Figueiredo Prudencio, R., Maximo, M. R. O. A., & Colombini, E. L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250269 · Cited by 361
Du, H., Zhang, R., Liu, Y., Wang, J., Lin, Y., Li, Z., Niyato, D., Kang, J., Xiong, Z., Cui, S., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3400011 · Cited by 254
Dataset Distillation by Matching Training Trajectories
Cazenavette, G., Wang, T., Torralba, A., Efros, A. A., & Zhu, J.-Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01045 · Cited by 240
“Such methods (behavior cloning in particular) have been shown to work well in offline settings [9, 12].”
MethodA Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Current offline RL benchmarks (Fu et al., 2021; Gulcehre et al., 2021) do not measure generalisation in this way, but we believe they should enable us to tackle this combined problem: for example, training on offline data from 200 levels in OpenAI Procgen, and evaluating on the full distribution of levels.”
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Nair, A., Gupta, A., Dalal, M., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09359 · Cited by 175
Self-citationFlorence, P., Lynch, C., Zeng, A., Ramirez, O., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., & Tompson, J. (2021) · arXiv · doi:10.48550/arxiv.2109.00137 · Cited by 131
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119
“We also provide support for a simulation environment on the Franka-Kitchen benchmark [28].”
MethodRearrangement: A Challenge for Embodied AI
Batra, D., Chang, A. X., Chernova, S., Davison, A. J., Deng, J., Koltun, V., Levine, S., Malik, J., Mordatch, I., Mottaghi, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.01975 · Cited by 109
“…tasks focusing on standardized real-world object sets [13], and competition-style setups, such as RoboCup [43, 67] and the International Planning Competition [53], as well as simulated benchmarks focusing on specific algorithmic approaches such as reinforcement learning [5, 9, 24, 79, 26].”
BackgroundSelf-citationThe Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M. (., Arunachalam, S. P., & Pinto, L. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.010 · Cited by 95
The Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M., Arunachalam, S. P., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2112.01511 · Cited by 94
“Note that although we use taskspecific demonstrations for representation learning, our framework is compatible with using other forms of unlabelled data such offline datasets [16, 13] or task-agnostic play data [45].”
MethodAugmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
Wang, Z., Hunt, J. J., & Zhou, M. (2022) · arXiv · doi:10.48550/arxiv.2208.06193 · Cited by 83
“With N = 5 , the training time of our method is similar to that of CQL (Kumar et al., 2020).”
BackgroundMethodInfluentialWhat Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., & Martín-Martín, R. (2021) · arXiv · doi:10.48550/arxiv.2108.03298 · Cited by 66
“Benchmark datasets [18, 19] have also been proposed recently for batch (offline) reinforcement learning.”
BackgroundMethodInfluentialGenerating Code World Models with Large Language Models Guided by Monte Carlo Tree Search
Dainese, N., Merler, M., Alakuijala, M., & Marttinen, P. (2024) · arXiv · doi:10.48550/arxiv.2405.15383 · Cited by 38
“A possible solution could be breaking down the CWM into smaller functions that can be re-written individually by an LLM, to account for some changes in the environment, or modeling variable factors as arguments to the step function.”
BackgroundManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34
“Along with the rise of offline reinforcement learning, benchmarks like D4RL [35] have been proposed to measure the progress of offline reinforcement learning algorithms.”
MethodReinforcement learning and bandits for speech and language processing: Tutorial, review and outlook
Lin, B. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122254 · Cited by 28
Offline Meta-Reinforcement Learning with Online Self-Supervision
Pong, V. H., Nair, A., Smith, L., Huang, C., & Levine, S. (2021) · arXiv · doi:10.48550/arxiv.2107.03974 · Cited by 25
Self-citationScaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24
“…aspects: First, how to scale up the data collection for a diverse range of manipulation skills, which involves efforts such as improving the hardware [1, 2] and software [3, 4] which support demonstration collection, utilization of non-robotics datasets [5, 6], or trial-and-error explorations [7].”
BackgroundMethodRoboCLIP: One Demonstration is Enough to Learn Robot Policies
Sontakke, S. A., Zhang, J., Arnold, S. M. R., Pertsch, K., Bıyık, E., Sadigh, D., Finn, C., & Itti, L. (2023) · arXiv · doi:10.48550/arxiv.2310.07899 · Cited by 23
C-Learning: Learning to Achieve Goals via Recursive Classification
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2011.08909 · Cited by 18
“For these experiments, we used the “expert” data provided for each task in Fu et al. (2020).”
MethodSelf-citationRL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
Yuan, Z., Yang, S., Hua, P., Chang, C., Hu, K., & Xu, H. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0295 · Cited by 10
Measuring Progress in Deep Reinforcement Learning Sample Efficiency
Dorner, F. E. (2021) · arXiv · doi:10.48550/arxiv.2102.04881 · Cited by 6
“As it might be easier to collect data produced by a human operator or an inefficient but save heuristic rather than an untrained agent, offline learning could play an important role for many real world applications of RL 12 (Agarwal et al., 2020; Fu et al., 2020).”
BackgroundS4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
Sinha, S., Mandlekar, A., & Garg, A. (2021) · arXiv · doi:10.48550/arxiv.2103.06326 · Cited by 5
XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX
Nikulin, A., Kurenkov, V., Zisman, I., Agarkov, A., Sinii, V., & Kolesnikov, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1390 · Cited by 3
“Similar to Fu et al. (2020); Kurenkov et al. (2023), our benchmarks are hosted in the cloud and will be downloaded and cached the first time they are used.”
BackgroundJackson, M. T., Matthews, M. T., Lu, C., Ellis, B., Whiteson, S., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2404.06356 · Cited by 2
Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Sobal, V., Zhang, W., Cho, K., Balestriero, R., Rudner, T. G. J., & LeCun, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.14819 · Cited by 1
Metadata
- Authors
- Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2004.07219
- OpenAlex
- W3016525976
- Cited by
- 310
- References
- 44
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.