Distributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids
Open accessAbstract
The power consumption of households has been constantly growing over the years. To cope with this growth, intelligent management of the consumption profile of the households is necessary, such that the households can save the electricity bills, and the stress to the power grid during peak hours can be reduced. However, implementing such a method is challenging due to the existence of randomness in the electricity price and the consumption of the appliances. To address this challenge, in this article, we employ a model-free method for the households, which works with limited information about the uncertain factors. More specifically, the interactions between households and the power grid can be modeled as a noncooperative stochastic game, where the electricity price is viewed as a stochastic variable. To search for the Nash equilibrium (NE) of the game, we adopt a method based on distributed deep reinforcement learning. Also, the proposed method can preserve the privacy of the households. We then utilize real-world data from Pecan Street Inc., which contains the power consumption profile of more than 1000 households, to evaluate the performance of the proposed method. In average, the results reveal that we can achieve around 12% reduction on peak-to-average ratio and 11% reduction on load variance. With this approach, the operation cost of the power grid and the electricity cost of the households can be reduced.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DRL can be integrated into the FL framework for privacy-preserving learning [199], [216].
“Moreover, DRL can be integrated into the FL framework for privacy-preserving learning [199], [216].”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [216]
A distributed deep reinforcement learning method inspired by federated learning is proposed to preserve the privacy of households while intelligently managing their load scheduling, with action networks at distributed households and the critic network at an aggregator from a trusted third party.
“Furthermore, to preserve the privacy of the households while intelligently managing their load scheduling, a distributed deep reinforcement learning method is proposed inspired by the idea of federated learning [216], where the action networks are located at distributed households and the critic network is located at an aggregator from a trusted third party.”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [216]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
(1999) · Cited by 5,147 · see citing works
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275 · see citing works
Demand Side Management: Demand Response, Intelligent Energy Systems, and Smart Loads
Pálenský, P., & Dietrich, D. (2011) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2011.2158841 · Cited by 2,959 · see citing works
Multiagent systems: algorithmic, game-theoretic, and logical foundations
(2009) · Choice Reviews Online · doi:10.5860/choice.46-5662 · Cited by 1,756 · see citing works
Shoham, Y., & Leyton‐Brown, K. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511811654 · Cited by 979 · see citing works
Dependable Demand Response Management in the Smart Grid: A Stackelberg Game Approach
Maharjan, S., Zhu, Q., Zhang, Y., Gjessing, S., & Başar, T. (2013) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2012.2223766 · Cited by 784 · see citing works
Self-citationMultiagent Reinforcement Learning: Theoretical Framework and an Algorithm
Hu, J., & Wellman, M. P. (1998) · Cited by 759 · see citing works
An Optimal Power Scheduling Method for Demand Response in Home Energy Management System
Zhao, Z., Lee, W. C., Shin, Y., & Song, K.-B. (2013) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2013.2251018 · Cited by 708 · see citing works
On-Line Building Energy Optimization Using Deep Reinforcement Learning
Mocanu, E., Mocanu, D. C., Nguyen, P. H., Liotta, A., Webber, M. E., Gibescu, M., & Slootweg, J. G. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2834219 · Cited by 649 · see citing works
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Wan, Z., Li, H., He, H., & Prokhorov, D. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2879572 · Cited by 548 · cited 3× · see citing works
“To further address the discretization issue associated with Q-learning [18]–[23], DQN [24]–[26], and policy gradient [27], [28], the employed model-free approach is similar to model-based methods that provide a continuous value as the solution without discretizing the action space.”
MethodThe O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
Borkar, V. S., & Meyn, S. (2000) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012997331639 · Cited by 528 · see citing works
Fast gradient-descent methods for temporal-difference learning with linear function approximation
Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., & Wiewiora, E. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553501 · Cited by 500 · see citing works
Lu, R., Hong, S. H., & Yu, M. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2909266 · Cited by 390 · see citing works
Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning
Ruelens, F., Claessens, B., Vandael, S., Schutter, B. D., Babuška, R., & Belmans, R. J. M. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2517211 · Cited by 339 · see citing works
Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning
Kim, B., Zhang, Y., van der Schaar, M., & Lee, J.-W. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2495145 · Cited by 299 · see citing works
Self-citationOptimal Demand Response Using Device-Based Reinforcement Learning
Wen, Z., O’Neill, D. C., & Maei, H. R. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2396993 · Cited by 271 · see citing works
Reinforcement Learning to Play an Optimal Nash Equilibrium in Team Markov Games
Wang, X., & Sandholm, T. (2002) · Cited by 251 · see citing works
A Distributed Algorithm for Managing Residential Demand Response in Smart Grids
Safdarian, A., Fotuhi‐Firuzabad, M., & Lehtonen, M. (2014) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2014.2316639 · Cited by 241 · see citing works
Demand Response Management in the Smart Grid in a Large Population Regime
Maharjan, S., Zhu, Q., Zhang, Y., et al. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2431324 · Cited by 224 · see citing works
Self-citationReinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem
Ernst, D., Glavić, M., Capitanescu, F., & Wehenkel, L. A. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.2007630 · Cited by 214 · see citing works
Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice
Ruelens, F., Claessens, B., Quaiyum, S., Schutter, B. D., Babuška, R., & Belmans, R. J. M. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2640184 · Cited by 174 · see citing works
Real-Time Demand Side Management for a Microgrid Considering Uncertainties
Yang, X., Zhang, Y., He, H., et al. (2018) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2825388 · Cited by 152 · see citing works
Self-citationA Decentralized Electricity Market Scheme Enabling Demand Response Deployment
Bahrami, S., Amini, M. H., Shafie‐khah, M., & Catalào, J. P. S. (2017) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2017.2771279 · Cited by 149 · see citing works
An Online Learning Algorithm for Demand Response in Smart Grid
Bahrami, S., Wong, V. W. S., & Huang, J. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2667599 · Cited by 147 · see citing works
Smart Home Energy Management Including Renewable Sources: A QoE-driven Approach
Pilloni, V., Floris, A., Meloni, A., & Atzori, L. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2605182 · Cited by 131 · see citing works
Optimal Industrial Load Control in Smart Grid
Gholian, A., Mohsenian‐Rad, H., & Hua, Y. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2468577 · Cited by 111 · see citing works
Gennitsaris, S., & Kanellos, F. D. (2019) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2919949 · Cited by 93 · see citing works
Prosumer Community: A Risk Aversion Energy Sharing Model
Cui, S., Wang, Y., Li, C., et al. (2019) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2019.2909301 · Cited by 88 · see citing works
Energy-Efficient Thermal Comfort Control in Smart Buildings via Deep Reinforcement LearningPreprint
Gao, G., Li, J., & Wen, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04693 · Cited by 71 · see citing works
Rassaei, F., Soh, W.-S., & Chua, K.-C. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2629515 · Cited by 67 · see citing works
Learning-Based Demand Response for Privacy-Preserving Users
Ghasemkhani, A., Yang, L., & Zhang, J. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2898462 · Cited by 48 · see citing works
Achieving Privacy Protection Using Distributed Load Scheduling: A Randomized Approach
Liu, E., & Cheng, P. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2703400 · Cited by 44 · see citing works
Distributed Demand Peak Reduction With Non-Cooperative Players and Minimal Communication
Collins, L. D., & Middleton, R. H. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2017.2734113 · Cited by 33 · see citing works
An Incentive-Compatible Scheme for Electricity Cooperatives: An Axiomatic Approach
Ehsanfar, A., & Heydari, B. (2016) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2591507 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
- Authors
- Chung, H.-M., Maharjan, S., Zhang, Y., & Eliassen, F.
- Affiliations
- University of Oslo (NO)
- Venue
- IEEE Transactions on Industrial Informatics
- Published
- 2021-04
- Volume
- 17
- Issue
- 4
- Pages
- 2752-2763
- ISSN
- 1551-3203 (ISSN-L)
- 1941-0050
- OpenAlex
- W3037739259
- Cited by
- 120
- References
- 38
- Citations by year
- 2026: 14 · 2025: 23 · 2024: 17 · 2023: 17 · 2022: 29 · 2021: 17 · 2020: 3
- FWCI
- 5.67
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Industrial Informatics (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Norges Forskningsråd (275106, 287412, 267967)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Smart Grid Energy Management
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Electric Vehicles and Infrastructure
- Physical Sciences › Engineering › Electrical and Electronic Engineering › IoT-based Smart Home Systems
- Keywords
- load scheduling, distributed deep reinforcement learning, Nash equilibrium, electricity price uncertainty, privacy preservation, peak-to-average power ratio reduction, household electricity costs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
