Investigating the Practicality of Existing Reinforcement Learning Algorithms: A Performance Comparison
Open accessThe DOI matches its Crossref record; 65 of the 66 references it cites were found in a registry or an index.
Abstract
Abstract---Reinforcement learning (RL) has become more popular due to promising results in applications such as chat-bots, healthcare, and autonomous driving. However, one significant challenge in current RL research is the difficulty in understanding which RL algorithms, if any, are practical for a given use case. Few RL algorithms are rigorously tested, and hence understood, for their practical implications. Although there are a number of performance comparisons in literature, many use few environments and do not consider real-world limitations such as run-time and memory usage. Furthermore, many works do not make their code publicly accessible for others to use. This paper addresses this gap by presenting the most comprehensive performance comparison on the practicality of RL algorithms known to date. Specifically, this paper focuses on discrete, model-free deep RL algorithms for their practicality in real-world problems where efficient implementations are necessary. In total, fourteen RL algorithms were trained on twenty-three environments (468 environment instances), which collectively required 224 GB and 766 days CPU time to run all experiments, and 1.7 GB to store all models. Overall, the results indicate several shortcomings in RL algorithms' exploration efficiency, memory/sample efficiency, and space/time complexity. Based on these shortcomings, numerous opportunities for future works were identified to improve the capabilities of modern algorithms. This paper’s findings will help researchers and practitioners improve and employ RL algorithms in time-sensitive and resource-constrained applications such as economics, cybersecurity, and Internet of Things (IoT). Impact Statement---Reinforcement learning (RL) technologies are commonly used in autonomous driving, chat-bot, and business analytic applications. They learn how to adapt to unforeseen situations, reducing the load on human drivers, support teams, and analysts. Although there are a variety of theoretical works in RL literature, very few algorithms are tested and evaluated to facilitate their use in real-life scenarios. The performance comparison introduced in this paper addresses these limitations. The performance analysis framework, re-implemented source code, and findings identified in this study could increase the adoption and speed development of RL technologies in more real-life applications. Moreover, the open challenges, recommendations, and practical implications identified in this paper could facilitate collaboration and development of new technologies among researchers and practitioners in industry and academia.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dreamer establishes a state of the art on this benchmark, outperforming Boot DQN and other methods46.”
From Mastering diverse control tasks through world models (2025) · cites this paper as [46]
In the citing paper
“Dreamer establishes a new state-of-the-art on this benchmark, outperforming Boot DQN and other methods 49 .”
From Mastering Diverse Domains through World Models (2023) · cites this paper as [49]
Citations
Cites
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
(1999) · Cited by 5,147
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Stable-Baselines3: Reliable Reinforcement Learning Implementations
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N. (2021) · elib (German Aerospace Center) · Cited by 1,000
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Reinforcement Learning · doi:10.1007/978-1-4615-3618-5_2 · Cited by 903
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Kakade, S. M. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 775
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 704
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645
Reinforcement Learning in Healthcare: A Survey
Yu, C., Liu, J., Nemati, S., & Yin, G. (2023) · ACM Computing Surveys · doi:10.1145/3477600 · Cited by 628
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497
RLlib: Abstractions for Distributed Reinforcement Learning
Liang, E., Liaw, R., Moritz, P., Nishihara, R., Fox, R., Goldberg, K., Gonzalez, J. E., Jordan, M. I., & Stoica, I. (2017) · arXiv · doi:10.48550/arxiv.1712.09381 · Cited by 473
Deep Exploration via Bootstrapped DQN
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 434
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359
Deep Exploration via Randomized Value FunctionsPreprint
Osband, I., Russo, D., Wen, Z., & Roy, B. V. (2017) · Journal of machine learning research · doi:10.48550/arxiv.1703.07608 · Cited by 350
Implicit Quantile Networks for Distributional Reinforcement Learning
Dabney, W., Ostrovski, G., Silver, D., & Munos, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06923 · Cited by 331
Explainability in deep reinforcement learning
Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106685 · Cited by 319
Deep Reinforcement Learning: An Overview
Mousavi, S. S., Schukat, M., & Howley, E. (2018) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-319-56991-8_32 · Cited by 261
Explainable Reinforcement Learning: A Survey
Puiutta, E., & Veith, E. M. S. P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_5 · Cited by 252
Reinforcement learning in urban network traffic signal control: A systematic literature review
Noaeen, M., Naik, A., Goodman, L., Crebo, J., Abrar, T., Abad, Z. S. H., Bazzan, A. L. C., & Far, B. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116830 · Cited by 246
A Distributional Perspective on Reinforcement Learning
Bellemare, M. G., Dabney, W., & Munos, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06887 · Cited by 237
Benchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226
Behaviour Suite for Reinforcement LearningPreprint
Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepesvári, C., Singh, S. P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03568 · Cited by 216
Practical Reinforcement Learning in Continuous Spaces
Smart, W. D., & Kaelbling, L. P. (2000) · Cited by 208
Soft Actor-Critic for Discrete Action Settings
Christodoulou, P. (2019) · arXiv · doi:10.48550/arxiv.1910.07207 · Cited by 204
Leveraging Procedural Generation to Benchmark Reinforcement Learning
Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2019) · arXiv · doi:10.48550/arxiv.1912.01588 · Cited by 196
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2012) · arXiv · doi:10.48550/arxiv.1207.4708 · Cited by 187
Reinforcement learning applied to production planning and control
Esteso, A., Peidro, D., Mula, J., & Díaz‐Madroñero, M. (2022) · International Journal of Production Research · doi:10.1080/00207543.2022.2104180 · Cited by 180
Survey on reinforcement learning for language processing
Uc-Cetina, V., Navarro-Guerrero, N., Martin-Gonzalez, A., Weber, C., & Wermter, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10205-5 · Cited by 177
SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark
Fan, L., Zhu, Y., Zhu, J., Liu, Z., Zeng, O., Gupta, A., Creus‐Costa, J., Savarese, S., & Fei-Fei, L. (2018) · Conference on Robot Learning · Cited by 176
Zhao, X., Xia, L., Tang, J., & Yin, D. (2019) · ACM SIGWEB Newsletter · doi:10.1145/3320496.3320500 · Cited by 165
Fully Parameterized Quantile Function for Distributional Reinforcement LearningPreprint
Yang, D., Li, Z., Lin, Z., Qin, T., Bian, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02140 · Cited by 164
Explainable Deep Reinforcement Learning: State of the Art and Challenges
Vouros, G. A. (2023) · ACM Computing Surveys · doi:10.1145/3527448 · Cited by 160
Tran‐Dang, H., Bhardwaj, S., Rahim, T., Musaddiq, A., & Kim, D. (2022) · Journal of Communications and Networks · doi:10.23919/jcn.2021.000041 · Cited by 156
Flow: Architecture and Benchmarking for Reinforcement Learning in Traffic Control.Preprint
Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., & Bayen, A. M. (2017) · arXiv (Cornell University) · Cited by 149
Benchmarking Batch Deep Reinforcement Learning Algorithms
Fujimoto, S., Conti, E., Ghavamzadeh, M., & Pineau, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01708 · Cited by 143
A Survey of Deep Reinforcement Learning in Video Games
Shao, K., Tang, Z., Zhu, Y., Li, N., & Zhao, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10944 · Cited by 141
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
Lin, X., Wang, Y., Olkin, J., & Held, D. (2020) · arXiv · doi:10.48550/arxiv.2011.07215 · Cited by 140
Randomized Prior Functions for Deep Reinforcement Learning
Osband, I., Aslanides, J., & Cassirer, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03335 · Cited by 103
Machine learning in games: a survey
Fürnkranz, J. (2001) · Nova Science Publishers eBooks · Cited by 82
Reinforcement learning for personalization: A systematic literature review
den Hengst, F., Grua, E. M., el Hassouni, A., & Hoogendoorn, M. (2020) · Data Science · doi:10.3233/ds-200028 · Cited by 82
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
Reinforcement learning in financial markets - a surveyPreprint
Fischer, T. (2018) · RePEc: Research Papers in Economics · Cited by 69
A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue ManagementPreprint
Casanueva, I., Budzianowski, P., Su, P.-H., Mrkšić, N., Wen, T.-H., Ultes, S., Rojas-Barahona, L. M., Young, S. J., & Gašić, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11023 · Cited by 62
LIFT: Reinforcement Learning in Computer Systems by Learning From DemonstrationsPreprint
Schaarschmidt, M., Kuhnle, A., Ellis, B., Fricke, K., Gessert, F., & Yoneki, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07903 · Cited by 59
Benchmarking Reinforcement Learning Algorithms on Real-World Robots
Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., & Bergstra, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07731 · Cited by 45
Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and BenchmarkingPreprint
Pardo, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07537 · Cited by 40
The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
Stone, A., Ramirez, O., Konolige, K., & Jonschkowski, R. (2021) · arXiv · doi:10.48550/arxiv.2101.02722 · Cited by 31
Acme: A Research Framework for Distributed Reinforcement Learning
Hoffman, M. W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Stańczyk, P., Ramos, S., Raichuk, A., Vincent, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.00979
Tianshou: a Highly Modularized Deep Reinforcement Learning Library
Weng, J., Chen, H., Yan, D., You, K., Duburcq, A., Zhang, M., Su, Y., Su, H., & Zhu, J. (2021) · arXiv · doi:10.48550/arxiv.2107.14171
Cited by
Mastering diverse control tasks through world models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025) · Nature · doi:10.1038/s41586-025-08744-2 · Cited by 189
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124
Metadata
- Authors
- Dizon-Paradis, O., Wormald, S., Capecci, D., Bhandarkar, A., & Woodard, D.
- Affiliations
- University of Florida (US)
- Corresponding author
- Woodard, Damon
- Published
- 2023-07-30
- OpenAlex
- W4385396896
- Cited by
- 1
- References
- 66
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- reinforcement learning algorithms, performance comparison, exploration efficiency, memory efficiency, sample efficiency, computational complexity, resource-constrained applications, Internet of Things, open-source code
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.