Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
In reinforcement learning (RL), function approximation errors are known to easily lead to the Q -value overestimations, thus greatly reducing policy performance. This article presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating Q -value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate Q -value overestimations because it is capable of adaptively adjusting the update step size of the Q -value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 4×
“Deep RL algorithms, such as deep Q-networks (DQNs) [1], employ a deep NN to estimate the Q value.”
BackgroundMethodInfluentialReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 4×
“While many prior RL algorithms consider the policy entropy, they only use it as a regularizer [3], [24], [25].”
BackgroundMethodInfluentialSoft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 4,896
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · UvA-DARE (University of Amsterdam) · Cited by 4,017
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · cited 16×
“We additionally compare our method with our proposed twin delayed distributional deep deterministic policy gradient (TD4) algorithm, which is developed by replacing the clipped double Q-learning in TD3 with the distributional return learning: double Q-learning variant of SAC (Double-Q SAC), in which we replace the cli…”
BackgroundMethodInfluentialGaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · cited 10×
“In this article, we consider a more general entropy-augmented objective [16], [17], [32], which augments the reward with a policy entropy term H”
BackgroundMethodInfluentialIMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.01561 · Cited by 1,708
Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear UnitsPreprint
Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · Cited by 1,156
Distributed Prioritized Experience Replay
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., & Silver, D. (2018) · arXiv · doi:10.48550/arxiv.1803.00933 · Cited by 963
van Hasselt, H. P. (2010) · Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands · Cited by 839
Emergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657 · cited 1×
“Over the last decade, numerous deep RL algorithms have appeared [1], [3], [12], [14], [16], [23]–[26].”
BackgroundDistributional Reinforcement Learning With Quantile Regression
Dabney, W., Rowland, M., Bellemare, M., & Munos, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11791 · Cited by 612
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582 · cited 1×
“To evaluate our algorithm, we measure its performance and Q-value estimation bias on a suite of MuJoCo continuous control tasks without modifications to environment [37], interfaced through OpenAI Gym [38].”
MethodDistributed Distributional Deterministic Policy Gradients
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., & Lillicrap, T. (2018) · arXiv · doi:10.48550/arxiv.1804.08617 · Cited by 567
A distributional code for value in dopamine-based reinforcement learning
Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., & Botvinick, M. (2020) · Nature · doi:10.1038/s41586-019-1924-6 · Cited by 443
Reinforcement Learning with Deep Energy-Based Policies
Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08165 · Cited by 392
Implicit Quantile Networks for Distributional Reinforcement Learning
Dabney, W., Ostrovski, G., Silver, D., & Munos, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06923 · Cited by 331
Issues in Using Function Approximation for Reinforcement Learning
Thrun, S., & Schwartz, A. (1999) · Cited by 311
Bridging the Gap Between Value and Policy Based Reinforcement Learning
Nachum, O., Norouzi, M., Xu, K., & Schuurmans, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08892 · Cited by 198
Duan, J., Eben Li, S., Guan, Y., Sun, Q., & Cheng, B. (2020) · IET Intelligent Transport Systems · doi:10.1049/iet-its.2019.0317 · Cited by 189
Self-citationEquivalence Between Policy Gradients and Soft Q-Learning
Schulman, J., Chen, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06440 · Cited by 171
An Analysis of Categorical Distributional Reinforcement LearningPreprint
Rowland, M., Bellemare, M., Dabney, W., Munos, R., & Teh, Y. (2018) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.1802.08163 · Cited by 138
Reinforcement Learning with Factored States and Actions
Sallans, B., & Hinton, G. E. (2004) · doi:10.5555/1005332.1016794 · Cited by 133
Distributional Reinforcement Learning with Quantile Regression
Dabney, W., Rowland, M., Bellemare, M. G., & Munos, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10044 · Cited by 128
A Comparative Analysis of Expected and Distributional Reinforcement Learning
Lyle, C., Bellemare, M. G., & Castro, P. S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014504 · Cited by 114
Combining policy gradient and Q-learning
O'Donoghue, B., Munos, R., Kavukcuoglu, K., & Mnih, V. (2016) · arXiv · doi:10.48550/arxiv.1611.01626 · Cited by 90 · cited 2×
“Recently, several papers have noted the connection between Q-learning and policy gradient methods in the setting of the maximum entropy framework [27]–[29].”
BackgroundCombining policy gradient and Q-learning
O’Donoghue, B., Munos, R., Kavukcuoglu, K., & Mnih, V. (2017) · International Conference on Learning Representations · Cited by 52
Bias-corrected Q-learning to control max-operator bias in Q-learning
Lee, D., Defourny, B., & Powell, W. B. (2013) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2013.6614994 · Cited by 44
Cited by
Li, G., Yang, Y., Li, S., Qu, X., Lyu, N., & Li, S. E. (2022) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2021.103452 · Cited by 308
Self-citationA Survey on Reinforcement Learning in Aviation Applications
Razzaghi, P., Tabrizian, A., Guo, W., Chen, S., Taye, A., Thompson, E., Bregeon, A., Baheri, A., & Wei, P. (2022) · arXiv · doi:10.48550/arxiv.2211.02147 · Cited by 86
Metadata
- Authors
- Duan, J., Guan, Y., Li, S. E., Ren, Y., Sun, Q., & Cheng, B.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2022-11
- Volume
- 33
- Issue
- 11
- Pages
- 6584-6598
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3015082424
- Cited by
- 325
- References
- 42
- FWCI
- 13.41
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.