The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems
Abstract
Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 4×
“By synergistically combining the merits of the method of multipliers and soft actor-critic (SAC) [31] algorithm, our proposed CSAC algorithm can better satisfy the operation constraints in power distribution systems.”
BackgroundMethodInfluentialTrust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · cited 2×
“To mitigate the positive bias in the policy update step, the clipped double Q-learning technique [41] is adopted.”
MethodDeep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Radial distribution test feeders
Kersting, W. H. (1991) · IEEE Transactions on Power Systems · doi:10.1109/59.119237 · Cited by 1,888
Radial distribution test feeders
Kersting, W. H. (2002) · 2001 IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No.01CH37194) · doi:10.1109/pesw.2001.916993 · Cited by 1,051
Antoniadou-Plytaria, K. E., Kouveliotis-Lysikatos, I. N., Georgilakis, P. S., & Hatziargyriou, N. D. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2679238 · Cited by 545 · cited 1×
“…communication delay issue of the centralized optimization and control approach for VVC, the third group of literature develops non-centralized control schemes, which can be further divided into three subgroups, local VVC algorithms, distributed VVC algorithms, and decentralized VCC algorithms [14].”
MethodLi, P., Ji, H., Wang, C., Zhao, J., Song, G., Ding, F., & Wu, J. (2017) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2017.2686009 · Cited by 447
Adaptive Power System Emergency Control Using Deep Reinforcement Learning
Huang, Q., Huang, R., Hao, W., Tan, J., Fan, R., & Huang, Z. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2933191 · Cited by 415
Fast Local Voltage Control Under Limited Reactive Power: Optimality and Stability Analysis
Zhu, H., & Liu, H. J. (2016) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2015.2504419 · Cited by 409
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · UvA-DARE (University of Amsterdam) · Cited by 407
Reinforcement Learning with Deep Energy-Based Policies
Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08165 · Cited by 392
Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy
Ziebart, B. D. (2010) · doi:10.1184/r1/6720692 · Cited by 380
Off-Policy Actor-CriticPreprint
Degris, T., White, M., & Sutton, R. S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.4839 · Cited by 378
Multi-Timescale Coordinated Voltage/Var Control of High Renewable-Penetrated Distribution Systems
Xu, Y., Dong, Z. Y., Zhang, R., & Hill, D. J. (2017) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2017.2669343 · Cited by 340
An Optimal and Distributed Method for Voltage Regulation in Power Distribution Systems
Zhang, B., Lam, A. Y. S., Dominguez-Garcia, A. D., & Tse, D. (2015) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2347281 · Cited by 325
Distributed Reactive Power Feedback Control for Voltage Regulation and Loss Minimization
Bolognani, S., Carli, R., Cavraro, G., & Zampieri, S. (2015) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2363931 · Cited by 252
Reward Constrained Policy Optimization
Tessler, C., Mankowitz, D. J., & Mannor, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11074 · Cited by 237
Reactive Power and Voltage Control in Distribution Systems With Limited Switching Operations
Liu, M. B., Canizares, C. A., & Huang, W. (2009) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2009.2016362 · Cited by 222
Bridging the Gap Between Value and Policy Based Reinforcement Learning
Nachum, O., Norouzi, M., Xu, K., & Schuurmans, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08892 · Cited by 198
Wang, Z., Wang, J., Chen, B., Begovic, M. M., & He, Y. (2014) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2329842 · Cited by 183
A Framework for Volt-VAR Optimization in Distribution Systems
Ahmadi, H., Marti, J. R., & Dommel, H. W. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2374613 · Cited by 183
Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning
Xu, H., Domínguez-García, A. D., & Sauer, P. W. (2019) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2948132 · Cited by 180
Discretizing Continuous Action Space for On-Policy Optimization
Tang, Y., & Agrawal, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6059 · Cited by 179
An actor-critic algorithm for constrained Markov decision processes
Borkar, V. S. (2005) · Systems & Control Letters · doi:10.1016/j.sysconle.2004.08.007 · Cited by 172
Optimal Distributed Voltage Regulation for Secondary Networks With DGs
Yu, L., Czarkowski, D., & de León, F. (2012) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2012.2190308 · Cited by 172
Distributed Voltage Control in Distribution Networks: Online and Robust Implementations
Liu, H. J., Shi, W., & Zhu, H. (2018) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2703642 · Cited by 163
Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch
Xu, Y., Zhang, W., Liu, W., & Ferrese, F. T. (2012) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2012.2218596 · Cited by 149
Reinforcement Learning for Reactive Power Control
Vlachogiannis, J. G., & Hatziargyriou, N. D. (2004) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2004.831259 · Cited by 138
Nazir, F. U., Pal, B. C., & Jabr, R. A. (2018) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2018.2859759 · Cited by 116
Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning
Diao, R., Wang, Z., Shi, D., Chang, Q., Duan, J., & Zhang, X. (2019) · 2019 IEEE Power & Energy Society General Meeting (PESGM) · doi:10.1109/pesgm40551.2019.8973924 · Cited by 115 · cited 1×
“It has been shown that the deep reinforcement learning (DRL) approach could overcome this problem in emergency system control [4], [5].”
BackgroundPhase Identification in Electric Power Distribution Systems by Clustering of Smart Meter Data
Wang, W., Yu, N., Foggo, B., Davis, J., & Li, J. (2016) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2016.0050 · Cited by 114
Self-citationCentralized Support Distributed Voltage Control by Using End-Users as Reactive Power Support
Abessi, A., Vahidinasab, V., & Ghazizadeh, M. S. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2410780 · Cited by 106
Tushar, M. H. K., & Assi, C. M. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2648509 · Cited by 74
Dynamic Reactive Power Control of Islanded Microgrids
Falahi, M., Butler-Purry, K. L., & Ehsani, M. (2013) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2013.2246589 · Cited by 68
Zheng, W., Wu, W., Zhang, B., & Wang, Y. (2017) · IET Generation Transmission & Distribution · doi:10.1049/iet-gtd.2016.0950 · Cited by 65
Improving Supervised Phase Identification Through the Theory of Information Losses
Foggo, B., & Yu, N. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2952080 · Cited by 50
Self-citationDynamic Distribution Network Reconfiguration Using Reinforcement Learning
Gao, Y., Shi, J., Wei, W., et al. (2019) · IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) · doi:10.1109/smartgridcomm.2019.8909777 · Cited by 44
Self-citationVolt-VAR Control in Power Distribution Systems with Deep Reinforcement Learning
Wang, W., Yu, N., Shi, J., & Gao, Y. (2019) · IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) · doi:10.1109/smartgridcomm.2019.8909741 · Cited by 38
Self-citation
Cited by
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review
Cao, D., Hu, W., Zhao, J., Zhang, G., Zhang, B., Liu, Z., Chen, Z., & Blaabjerg, F. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000552 · Cited by 460
Chen, X., Qu, G., Tang, Y., Low, S., & Li, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3154718 · Cited by 415
Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems
Zhang, Y., Wang, X., Wang, J., & Zhang, Y. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3010130 · Cited by 299
Multi-Agent Safe Policy Learning for Power Management of Networked Microgrids
Zhang, Q., Dehghanpour, K., Wang, Z., Qiu, F., & Zhao, D. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3034827 · Cited by 137
Cao, D., Zhao, J., Hu, J., Pei, Y., Huang, Q., Chen, Z., & Hu, W. (2024) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2023.3267069 · Cited by 103
“Moreover, the SP and RO methods are model-based that are vulnerable to system model errors, commonly seen in practice [8], [9].”
Method
Metadata
- Authors
- Wang, W., Yu, N., Gao, Y., & Shi, J.
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2020-07
- Volume
- 11
- Issue
- 4
- Pages
- 3008-3018
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2997572754
- Cited by
- 331
- References
- 44
- FWCI
- 9.60
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.