A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Open accessThe DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
(1995) · Cited by 10,558
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 1×
“Regarding policy regularization, Haarnoja et al. [54] proposed an entropy regularization term in their seminal work on soft actor–critics (SACs).”
BackgroundDeep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 2×
“These quantities are used throughout the RL field, which is conventionally subdivided into three classes of methods: dynamic programming, model free, and model based [31], [32].”
BackgroundA Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · cited 2×
“The D4RL benchmark includes datasets for OpenAI’s Gym-MuJoCo tasks, mazes, dexterous manipulation tasks (i.e., Adroit [68]), robotic manipulation tasks (i.e., FrankaKitchen [69]), autonomous driving (i.e., car learning to act (CARLA) [70]), and traffic simulation (i.e., flow [71]).”
BackgroundAddressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 1×
“We can also run trajectory optimization or planning under the learned model with methods like linear-quadratic regulator (LQR) [59] and Monte Carlo tree search (MCTS) [60].”
MethodDeep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 1×
“ronment [74] (ALE), and real-world RL suite [75].”
BackgroundOffline Reinforcement Learning with Implicit Q-LearningPreprint
Kostrikov, I., Nair, A., & Levine, S. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2110.06169 · Cited by 1,934
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Zhang, K., Yang, Z., & Başar, T. (2021) · Studies in Systems, Decision and Control · doi:10.1007/978-3-030-60990-0_12 · Cited by 1,264
Deep Reinforcement Learning: A Survey
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., & Miao, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3207346 · Cited by 972
Review - Mastering the game of Go with deep neural networks and tree search
Babbar, S. (2017) · Unpublished · doi:10.13140/rg.2.2.18893.74727 · Cited by 929
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1×
“With these modified reward functions in hand, we can then use classic model-based approaches to tackle offline RL problems, such as Dyna-based methods that sample transitions from the model to train a model-free algorithm [19], [34].”
MethodSynthesis and stabilization of complex behaviors through online trajectory optimization
Tassa, Y., Erez, T., & Todorov, E. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386025 · Cited by 744
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · cited 1×
“Suite [72], DeepMind locomotion [73], arcade learning envi-”
BackgroundReinforcement learning in robotic applications: a comprehensive survey
Singh, B., Kumar, R., & Singh, V. P. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09997-9 · Cited by 392
When to Trust Your Model: Model-Based Policy Optimization
Janner, M., Fu, J., Zhang, M., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.08253 · Cited by 351 · cited 2×
“and favor decisions that are consistent across the models [19],”
BackgroundMethodD4RL: Datasets for Deep Data-Driven Reinforcement Learning
Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2004.07219 · Cited by 310
Information Theory and the Central Limit Theorem
Johnson, O. (2004) · IMPERIAL COLLEGE PRESS eBooks · doi:10.1142/p341 · Cited by 264
Maximum a Posteriori Policy Optimisation
Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., & Riedmiller, M. (2018) · arXiv · doi:10.48550/arxiv.1806.06920 · Cited by 259 · cited 1×
“to performing maximum a posteriori policy optimization [63] (MPO).”
MethodAlgaeDICE: Policy Gradient from Arbitrary ExperiencePreprint
Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., & Schuurmans, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02074 · Cited by 255
Behavior Regularized Offline Reinforcement Learning
Wu, Y., Tucker, G., & Nachum, O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11361 · Cited by 231
AlphaStar: an evolutionary computation perspective
Arulkumaran, K., Cully, A., & Togelius, J. (2019) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3319619.3321894 · Cited by 228
Reinforcement learning for clinical decision support in critical care: comprehensive review
Liu, S., See, K. C., Ngiam, K. Y., Celi, L. A., Sun, X., & Feng, M. (2020) · Journal of Medical Internet Research · doi:10.2196/18477 · Cited by 225 · cited 2×
“driving [12], [81], healthcare [11], [82], dialog systems [83],”
Backgrounddm_control: Software and tasks for continuous control
Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., & Tassa, Y. (2020) · Software Impacts · doi:10.1016/j.simpa.2020.100022 · Cited by 199
Hyperparameter Selection for Offline Reinforcement LearningPreprint
Paine, T. L., Păduraru, C., Michi, A., Gülçehre, Ç., Żołna, K., Novikov, A., Wang, Z., & de Freitas, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.09055 · Cited by 196
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Nair, A., Gupta, A., Dalal, M., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09359 · Cited by 175 · cited 3×
“Interestingly, Jiang and Li [39] do not attempt to correct this bias, likely because their advantage estimate would still be biased due to its reliance on a bootstrapped return.”
BackgroundMethodInfluentialSafe Reinforcement Learning With Stability Guarantee for Motion Planning of Autonomous Vehicles
Zhang, L., Zhang, R., Wu, T., Weng, R., Han, M., & Zhao, Y. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084685 · Cited by 165
Empirical Study of Off-Policy Policy Evaluation for Reinforcement LearningPreprint
Voloshin, C., Le, H. M., Jiang, N., & Yue, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06854 · Cited by 161
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00177 · Cited by 158
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
Gupta, A., Kumar, V., Lynch, C., Levine, S., & Hausman, K. (2019) · arXiv · doi:10.48550/arxiv.1910.11956 · Cited by 153 · cited 3×
“The D4RL benchmark includes datasets for OpenAI’s Gym-MuJoCo tasks, mazes, dexterous manipulation tasks (i.e., Adroit [68]), robotic manipulation tasks (i.e., FrankaKitchen [69]), autonomous driving (i.e., car learning to act (CARLA) [70]), and traffic simulation (i.e., flow [71]).”
Flow: Architecture and Benchmarking for Reinforcement Learning in Traffic Control.Preprint
Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., & Bayen, A. M. (2017) · arXiv (Cornell University) · Cited by 149
High-Confidence Off-Policy Evaluation
Thomas, P., Theocharous, G., & Ghavamzadeh, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9541 · Cited by 145
Benchmarking Batch Deep Reinforcement Learning Algorithms
Fujimoto, S., Conti, E., Ghavamzadeh, M., & Pineau, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01708 · Cited by 143
Wang, Z., Novikov, A., Zolna, K., Springenberg, J. T., Reed, S., Shahriari, B., Siegel, N., Merel, J., Gulcehre, C., Heess, N., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.15134 · Cited by 128 · cited 4×
“Furthermore, Wang et al. [45] propose another method named critic regularized regression (CRR), which also uses the indicator function to aggressively filter out below-average actions like in (47), but opts for a more pessimistic advantage estimator where we change the expectation to a max operator in the AWAC advanta…”
BackgroundMethodInfluentialZhan, X., Xu, H., Zhang, Y., Zhu, X., Yin, H., & Zheng, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20393 · Cited by 118
Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement LearningPreprint
Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., & Pinto, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.13425 · Cited by 118
Offline reinforcement learning for safer blood glucose control in people with type 1 diabetes
Emerson, H., Guy, M., & McConville, R. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104376 · Cited by 116
A Survey of Sim-to-Real Transfer Techniques Applied to Reinforcement Learning for Bioinspired Robots
Zhu, W. D., Guo, X., Owaki, D., Kutsuzawa, K., & Hayashibe, M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3112718 · Cited by 112
COG: Connecting New Skills to Past Experience with Offline Reinforcement LearningPreprint
Singh, A., Yu, A., Yang, J., Zhang, J. M., Kumar, A., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14500 · Cited by 110
Scaling data-driven robotics with reward sketching and batch reinforcement learning
Cabi, S., Gómez Colmenarejo, S., Novikov, A., Konyushova, K., Reed, S., Jeong, R., Zolna, K., Aytar, Y., Budden, D., Vecerik, M., et al. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.076 · Cited by 89
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78 · cited 1×
“ronment [74] (ALE), and real-world RL suite [75].”
BackgroundIncremental Reinforcement Learning With Prioritized Sweeping for Dynamic Environments
Wang, Z., Chen, C., Li, H.-X., Dong, D., & Tarn, T.-J. (2019) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2019.2899365 · Cited by 73
CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
Verma, S. S., Fu, J., Yang, S., & Levine, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.332 · Cited by 64
Reinforcement Learning for Education: Opportunities and ChallengesPreprint
Singla, A. K., Rafferty, A. N., Radanović, G., & Heffernan, N. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08828 · Cited by 55
RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
Gulcehre, C., Wang, Z., Novikov, A., Paine, T. L., Colmenarejo, S. G., Zolna, K., Agarwal, R., Merel, J., Mankowitz, D., Paduraru, C., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.13888 · Cited by 42
Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning
Bai, C., Xiao, T., Zhu, Z.-F., Wang, L.-X., Zhou, F., Garg, A., He, B., Liu, P., & Wang, Z.-R. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3217189 · Cited by 20
Cited by
Autonomous driving system: A comprehensive survey
Zhao, J., Zhao, W., Deng, B., Wang, Z., Zhang, F., Zheng, W., Cao, W., Nan, J., Lian, Y., & Burke, A. F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122836 · Cited by 361
A Survey on Safety-Critical Driving Scenario Generation—A Methodological Perspective
Ding, W., Xu, C., Arief, M., Lin, H., Li, B., & Zhao, D. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3259322 · Cited by 264
Hua, M., Qi, X., Chen, D., Jiang, K., Liu, Z. E., Sun, H., Zhou, Q., & Xu, H. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3574280 · Cited by 88
Metadata
- Authors
- Figueiredo Prudencio, R., Maximo, M. R. O. A., & Colombini, E. L.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2024-08
- Volume
- 35
- Issue
- 8
- Pages
- 10237-10257
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W4360584316
- Cited by
- 361
- References
- 55
- FWCI
- 31.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.