Optimal Demand Response Using Device-Based Reinforcement Learning
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Demand response (DR) for residential and small commercial buildings is estimated to account for as much as 65% of the total energy savings potential of DR, and previous work shows that a fully automated energy management system (EMS) is a necessary prerequisite to DR in these areas. In this paper, we propose a novel EMS formulation for DR problems in these sectors. Specifically, we formulate a fully automated EMS’s rescheduling problem as a reinforcement learning (RL) problem, and argue that this RL problem can be approximately solved by decomposing it over device clusters. Compared with existing formulations, our new formulation does not require explicitly modeling the user’s dissatisfaction on job rescheduling, enables the EMS to self-initiate jobs, allows the user to initiate more flexible requests, and has a computational complexity linear in the number of device clusters. We also demonstrate the simulation results of applying Q-learning, one of the most popular and classical RL algorithms, to a representative example.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1995) · Cited by 10,558
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Barto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_588 · Cited by 950
Dynamic Pricing, Advanced Metering, and Demand Response in Electricity Markets
Borenstein, S., Jaske, M., & Rosenfeld, A. H. (2002) · eScholarship (California Digital Library) · Cited by 519
(1997) · An Introduction to Natural Computation · doi:10.7551/mitpress/3917.003.0017 · Cited by 492
Residential implementation of critical-peak pricing of electricity
Herter, K. (2007) · Energy Policy · doi:10.1016/j.enpol.2006.06.019 · Cited by 350
Residential Demand Response Using Reinforcement Learning
O'Neill, D., Levorato, M., Goldsmith, A., & Mitra, U. (2010) · 2010 First IEEE International Conference on Smart Grid Communications · doi:10.1109/smartgrid.2010.5622078 · Cited by 281
Self-citationQuantifying Customer Response to Dynamic Pricing
Faruqui, A., & George, S. (2005) · The Electricity Journal · doi:10.1016/j.tej.2005.04.005 · Cited by 261
Herter, K., McAuliffe, P., & Rosenfeld, A. (2007) · Energy · doi:10.1016/j.energy.2006.01.014 · Cited by 228
Generalization and Exploration via Randomized Value FunctionsPreprint
Osband, I., Roy, B. V., & Wen, Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.0635 · Cited by 194
Self-citationA Survey of Utility Experience with Real Time Pricing
Barbose, G., Goldman, C., & Neenan, B. (2004) · doi:10.2172/836966 · Cited by 188
Gradient Temporal-Difference Learning Algorithms
Maei, H. R. (2011) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r30q50 · Cited by 119
Self-citationAn Integrated Architecture for Demand Response Communications and Control
LeMay, M., Nelli, R., Gross, G., & Gunter, C. A. (2008) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2008.60 · Cited by 115
Development and evaluation of fully automated demand response in large facilities
Piette, M. A., USDOE. Assistant Secretary for Energy Efficiency and Renewable Energy. Office of the Building Technologies Program, Sezgen, O., Watson, D., Motegi, N., Shockman, C., & ten Hope, L. (2004) · doi:10.2172/840331 · Cited by 113
Maei, H. R., & Sutton, R. S. (2010) · Artificial General Intelligence · doi:10.2991/agi.2010.22 · Cited by 106
Self-citationIndustrial power demand response analysis for one-part real-time pricing
Roos, J. G., & Lane, I. E. (1998) · IEEE Transactions on Power Systems · doi:10.1109/59.651628 · Cited by 105
Ruelens, F., Claessens, B. J., Vandael, S., Iacovella, S., Vingerhoets, P., & Belmans, R. (2014) · 2014 Power Systems Computation Conference · doi:10.1109/pscc.2014.7038106 · Cited by 78
THE ROLE OF DEMAND RESPONSE IN ELECTRIC POWER MARKET DESIGN
Braithwait, S. D., Eakin, K., & Christensen, L. R. (2002) · Cited by 71
Smart finite state devices: A modeling framework for demand response technologies
Turitsyn, K. S., Backhaus, S., Ananyev, M., & Chertkov, M. (2011) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2011.6160649 · Cited by 32
Claessens, B., Vandael, S., Ruelens, F., & Hommelberg, M. P. F. (2012) · IEEE PES Innovative Smart Grid Technologies Europe · doi:10.1109/isgteurope.2012.6465679 · Cited by 27
Use of Approximate Dynamic Programming for Production Optimization
Wen, Z., Durlofsky, L. J., Roy, B. V., et al. (2011) · SPE Reservoir Simulation Symposium · doi:10.2118/141677-ms · Cited by 12
Self-citation
Cited by
Reinforcement learning for demand response: A review of algorithms and modeling techniques
Vázquez-Canteli, J. R., & Nagy, Z. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.11.002 · Cited by 883
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
“Another application is the Electricity Systems , especially the intelligent electricity networks , which can benefit from RL techniques for improved power-delivery decisions [179], [180] and active resource management [181].”
BackgroundDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
“• semi-supervised RL, e.g., Finn et al. (2017) • deep symbolic RL, Garnelo et al. (2016) • intrinsic motivation, e.g., Stadie et al. (2015); Kulkarni et al. (2016); Oudeyer et al. (2016) • hyperparameter learning, e.g. Andrychowicz et al. (2016) • information extraction, e.g., Narasimhan et al. (2016) • text games, e.…”
BackgroundInfluentialAntonopoulos, I., Robu, V., Couraud, B., Kirli, D., Norbu, S., Kiprakis, A., Flynn, D., Elizondo-Gonzalez, S., & Wattam, S. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.109899 · Cited by 656
Reinforcement learning for building controls: The opportunities and challenges
Wang, Z., & Hong, T. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115036 · Cited by 577
Metadata
- Authors
- Wen, Z., O'Neill, D., & Maei, H.
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2015-09
- Volume
- 6
- Issue
- 5
- Pages
- 2312-2324
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2024998154
- Cited by
- 266
- References
- 25
- FWCI
- 8.02
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.