Multi-Agent Reinforcement Learning for Decentralized Resilient Secondary Control of Energy Storage Systems Against DoS Attacks
Abstract
While distributed secondary controllers have been studied for multiple energy storage systems in islanded microgrids, information infrastructure has to be added for the extensive information transmission among these secondary controllers and the additional communication among distributed controllers is costly and increases the vulnerability surface to cyberattacks. In this work, a data-driven decentralized secondary control scheme is proposed for multiple heterogeneous battery energy storage systems (BESSs). The proposed secondary control scheme can achieve frequency regulation and the state-of-charge (SoC) balancing simultaneously for BESSs without requiring accurate BESS models. This scheme leverages an asynchronous advantage actor-critic (A3C) based multi-agent deep reinforcement learning (MA-DRL) algorithm where the centralized off-line learning with shared convolutional neural networks (CNN) is designed to maximize global rewards and ensure the performance of the entire system and a decentralized online execution mechanism is applied to each BESS. Furthermore, in view of possible denial-of-service (DoS) attack on local communication networks used for signal transfer between secondary controllers and remote sensors, a signal-to-interference-plus-noise ratio (SINR)-based dynamic and proactive event-triggered communication mechanism is proposed to alleviate the impact of DoS attacks and reduce the occupation of communication resources. Simulation results on a four-bus multiple BESS system show that the proposed decentralized secondary controller can achieve simultaneous frequency regulation and SoC balancing. Comparison results with other event-triggered mechanisms and MA-DRL algorithms show the A3C based MA-DRL algorithm with CNN can obtain a comparatively optimal policy through training and the designed event-triggered strategy can dynamically adapt the release frequency based on real-time SINR and significantly reduce the occupied network bandwidth and packet loss rate (PER) induced by DoS attacks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Prime MotorX's perception layer, weak authentication and misconfiguration in RTD user credentials are exploited by default password, DoS, and brute force attacks.
“In this scenario, the exploited vulnerabilities are weak Authentication and Misconfiguration.The launched attacks include Default Passwords attack [48], [49], DoS attack [50], [51], and Brute Force attack [52], [53], with the target being the User Credentials.”
From A Critical Overview of Industrial Internet of Things Security and Privacy Issues Using a Layer-Based Hacking Scenario (2023) · cites this paper as [49]
Cites
Bishop, O. (2007) · Routledge eBooks · doi:10.4324/9780080477527-64 · Cited by 25,031 · see citing works
Guerrero, J. M., Vasquez, J. C., Matas, J., de Vicuna, L. G., & Castilla, M. (2011) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2010.2066534 · Cited by 5,075 · see citing works
Markov Games as a Framework for Multi-Agent Reinforcement Learning
Littman, M. L. (1994) · International Conference on Machine Learning · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 3,406 · see citing works
A Delay System Method for Designing Event-Triggered Controllers of Networked Control Systems
Yue, D., Tian, E., & Han, Q. (2012) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2012.2206694 · Cited by 1,681 · see citing works
Distributed Secondary Control for Islanded Microgrids—A Novel Approach
Shafiee, Q., Guerrero, J. M., & Vásquez, J. C. (2013) · IEEE Transactions on Power Electronics · doi:10.1109/tpel.2013.2259506 · Cited by 1,087 · see citing works
Secondary Frequency and Voltage Control of Islanded Microgrids via Distributed Averaging
Simpson-Porco, J. W., Shafiee, Q., Dörfler, F., Vasquez, J. C., Guerrero, J. M., & Bullo, F. (2015) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2015.2436879 · Cited by 984 · see citing works
Microgrid Stability Definitions, Analysis, and Examples
Farrokhabadi, M., Cañizares, C. A., Simpson-Porco, J. W., Nasr‐Azadani, E., Fan, L., Mendoza-Araya, P. A., Tonkoski, R., Tamrakar, U., Hatziargyriou, N. D., Lagos, D., et al. (2019) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2925703 · Cited by 756 · see citing works
Lu, X., Sun, K., Guerrero, J. M., et al. (2013) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2013.2279374 · Cited by 750 · see citing works
Battery Energy Storage System (BESS) and Battery Management System (BMS) for Grid-Scale Applications
Lawder, M. T., Suthar, B., Northrop, P. W. C., De, S., Hoff, C. M., Leitermann, O., Crow, M. L., Santhanagopalan, S., & Subramanian, V. R. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2014.2317451 · Cited by 723 · see citing works
Hu, S., Yue, D., Xie, X., et al. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2861834 · Cited by 462 · see citing works
Peng, C., Li, J., & Fei, M. (2016) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2016.2634122 · Cited by 435 · see citing works
Adaptive Event-Triggering ${H}_{\infty }$ Load Frequency Control for Network-Based Power Systems
Peng, C., Zhang, J., & Yan, H. (2017) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2017.2726965 · Cited by 404 · see citing works
Novel Multiagent Based Load Restoration Algorithm for Microgrids
Xu, Y., & Liu, W. (2011) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2010.2099675 · Cited by 334 · see citing works
A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning
Wang, S., Duan, J., Shi, D., Xu, C., Li, H., Diao, R., & Wang, Z. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2020.2990179 · Cited by 318 · see citing works
Event-Based Formation Control for Nonlinear Multiagent Systems Under DoS Attacks
Tang, Y. S., Zhang, D., Shi, P. J., et al. (2020) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2020.2979936 · Cited by 268 · see citing works
Input-Based Event-Triggering Consensus of Multiagent Systems Under Denial-of-Service Attacks
Xu, Y., Fang, M., Wu, Z., et al. (2018) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2875250 · Cited by 265 · see citing works
Ding, L., Han, Q., Ning, B., & Yue, D. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2955739 · Cited by 259 · see citing works
Cao, D., Zhao, J., Hu, W., Ding, F., Huang, Q. P., Chen, Z., & Blaabjerg, F. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3072251 · Cited by 220 · see citing works
Distributed Resilient Control for Energy Storage Systems in Cyber–Physical Microgrids
Deng, C., Wang, Y., Wen, C., et al. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2981549 · Cited by 216 · see citing works
Guo, D., Tang, L., Zhang, X., & Liang, Y. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2020.3020400 · Cited by 215 · see citing works
Shin, M., Choi, D., & Kim, J. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2944183 · Cited by 201 · see citing works
Event-Triggered Sliding-Mode Control for Multi-Area Power Systems
Su, X., Liu, X., & Song, Y. (2017) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2017.2677357 · Cited by 198 · see citing works
Cao, D., Zhao, J., Hu, W., et al. (2021) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2021.3057090 · Cited by 183 · see citing works
Online Multi-Agent Reinforcement Learning for Decentralized Inverter-Based Volt-VAR Control
Liu, H., & Wu, W. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3060027 · Cited by 173 · see citing works
Stochastic Distributed Secondary Control for AC Microgrids via Event-Triggered Communication
Lai, J., Lu, X., Yu, X., et al. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2966691 · Cited by 163 · see citing works
Resilient H∞ Filtering for Event-Triggered Networked Systems Under Nonperiodic DoS Jamming Attacks
Hu, S., Yue, D., Chen, X., et al. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2896249 · Cited by 162 · see citing works
Pahasa, J., & Ngamroo, I. (2014) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2372038 · Cited by 143 · see citing works
Kamruzzaman, M., Duan, J., Shi, D., & Benidris, M. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3078446 · Cited by 141 · see citing works
Chen, G., & Guo, Z. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2785811 · Cited by 135 · see citing works
An Adaptive Resilient Load Frequency Controller for Smart Grids With DoS Attacks
Lu, K., Zeng, G.-Q., Luo, X., et al. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2020.2983565 · Cited by 111 · see citing works
Data-Driven False Data Injection Attacks Against Power Grids: A Random Matrix Approach
Lakshminarayana, S., Kammoun, A., Debbah, M. A., et al. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3011391 · Cited by 99 · see citing works
Resilient State Estimation of Cyber-Physical System With Multichannel Transmission Under DoS Attack
Yuan, H., Xia, Y., & Yang, H. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.2964586 · Cited by 83 · see citing works
Xing, L., Mishra, Y., Tian, Y., et al. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2916152 · Cited by 82 · see citing works
Dual-Consensus-Based Distributed Frequency Control for Multiple Energy Storage Systems
Xing, L., Mishra, Y., Tian, Y., et al. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2904075 · Cited by 64 · see citing works
Munir, M. S., Abedin, S. F., Tran, N. H. K., Han, Z., Huh, E., & Hong, C. S. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3049381 · Cited by 62 · see citing works
Kim, Y.-J., & Wang, J. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2620176 · Cited by 51 · see citing works
Enhancing Power System Cyber-Security With Systematic Two-Stage Detection Strategy
Li, X., & Hedman, K. W. (2019) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2942333 · Cited by 48 · see citing works
Wang, Y., Deng, C., Liu, D., Xu, Y. H., & Dai, J. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2020.3039530 · Cited by 41 · see citing works
A Statistical Approach for Detection of Denial of Service Attacks in Computer Networks
Amma, N. G. B., Selvakumar, S., & Velusamy, R. L. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3022799 · Cited by 28 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Grid integration impacts and control strategies for renewable based microgrid
Fazal, S., Enamul Haque, M., Taufiqul Arif, M., Gargoom, A., & Oo, A. M. T. (2023) · Sustainable Energy Technologies and Assessments · doi:10.1016/j.seta.2023.103069 · Cited by 74 · see citing works
Eyeleko, A. H., & Feng, T. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3308195 · Cited by 38 · see citing works
“The launched attacks include default password attack [47], [48], DoS attack [49], [50], and Brute force attack [51], [52], with the target being the user credentials.”
Background
- Authors
- Chen, P., Liu, S., Chen, B., & Yu, L.
- Affiliations
- Carleton University (CA)
- Zhejiang University of Technology (CN)
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2022-05
- Volume
- 13
- Issue
- 3
- Pages
- 1739-1750
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W4206787406
- Cited by
- 120
- References
- 39
- Citations by year
- 2026: 20 · 2025: 36 · 2024: 19 · 2023: 32 · 2022: 13
- FWCI
- 10.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Smart Grid (publishedVersion)
- Funders
- NSERC Discovery Grant
- National Natural Science Funds of China (62073292)
- Zhejiang Provincial Natural Science Foundation of China (LR20F030004)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Microgrid Control and Optimization
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Frequency Control in Power Systems
- Physical Sciences › Engineering › Control and Systems Engineering › Smart Grid Security and Resilience
- Keywords
- multi-agent deep reinforcement learning, asynchronous advantage actor-critic, battery energy storage, islanded microgrids, frequency regulation, state of charge balancing, denial-of-service attacks, cyber-physical security, event-triggered communication, communication resource reduction, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
