The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning
Open accessAbstract
Driven by recent advances in batch Reinforcement Learning (RL), this paper contributes to the application of batch RL to demand response. In contrast to conventional model-based approaches, batch RL techniques do not require a system identification step, making them more suitable for a large-scale implementation. This paper extends fitted Q-iteration, a standard batch RL technique, to the situation when a forecast of the exogenous data is provided. In general, batch RL techniques do not rely on expert knowledge about the system dynamics or the solution. However, if some expert knowledge is provided, it can be incorporated by using the proposed policy adjustment method. Finally, we tackle the challenge of finding an open-loop schedule required to participate in the day-ahead market. We propose a model-free Monte Carlo method that uses a metric based on the state-action value function or Q-function and we illustrate this method by finding the day-ahead schedule of a heat-pump thermostat. Our experiments show that batch RL techniques provide a valuable alternative to model-based controllers and that they can be used to construct both closed-loop and open-loop policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
On-line Q-learning Using Connectionist Systems
Rummery, G. A. (1994) · Cited by 1,477
Tree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1088690 · Cited by 1,352
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Cited by
Reinforcement learning for demand response: A review of algorithms and modeling techniques
Vázquez-Canteli, J. R., & Nagy, Z. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.11.002 · Cited by 883
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
“Electricity Systems, especially intelligent electricity networks, have been also built based on RL techniques for power-delivery decisions [176], [177] and active resource management [178].”
BackgroundDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Antonopoulos, I., Robu, V., Couraud, B., Kirli, D., Norbu, S., Kiprakis, A., Flynn, D., Elizondo-Gonzalez, S., & Wattam, S. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.109899 · Cited by 656
Reinforcement learning for building controls: The opportunities and challenges
Wang, Z., & Hong, T. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115036 · Cited by 577
Review on the research and practice of deep learning and reinforcement learning in smart grids
China Electric Power Research Institute, Zhang, D., Han, X., Taiyuan University of Technology, Deng, C., & China Electric Power Research Institute (2018) · CSEE Journal of Power and Energy Systems · doi:10.17775/cseejpes.2018.00520 · Cited by 561
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Wan, Z., Li, H., He, H., & Prokhorov, D. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2879572 · Cited by 548
A Review of Deep Reinforcement Learning for Smart Building Energy Management
Yu, L., Qin, S., Zhang, M., Shen, C., Jiang, T., & Guan, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3078462 · Cited by 476
Deep Reinforcement Learning for Smart Home Energy Management
Yu, L., Xie, W., Xie, D., Zou, Y., Zhang, D., Sun, Z., Zhang, L., Zhang, Y., & Jiang, T. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2957289 · Cited by 471
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review
Cao, D., Hu, W., Zhao, J., Zhang, G., Zhang, B., Liu, Z., Chen, Z., & Blaabjerg, F. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000552 · Cited by 460
A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach
Lu, R., Hong, S. H., & Zhang, X. (2018) · Applied Energy · doi:10.1016/j.apenergy.2018.03.072 · Cited by 430
A Multi-Agent Reinforcement Learning-Based Data-Driven Method for Home Energy Management
Xu, X., Jia, Y., Xu, Y., Xu, Z., Chai, S., & Lai, C. S. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.2971427 · Cited by 420
Incentive-based demand response for smart grid with reinforcement learning and deep neural network
Lu, R., & Hong, S. H. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.12.061 · Cited by 394
Lu, R., Hong, S. H., & Yu, M. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2909266 · Cited by 389
Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings
Yu, L., Sun, Y., Xu, Z., Shen, C., Yue, D., Jiang, T., & Guan, X. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3011739 · Cited by 353
Qian, T., Shao, C., Wang, X., & Shahidehpour, M. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2942593 · Cited by 262
A Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 257
Glavic, M., Fonteneau, R., & Ernst, D. (2017) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2017.08.1217 · Cited by 241
Artificial Intelligence Enabled Demand Response: Prospects and Challenges in Smart Grid Environment
Khan, M. A., Saleh, A. M., Waseem, M., & Sajjad, I. A. (2023) · IEEE Access · doi:10.1109/access.2022.3231444 · Cited by 179
Qian, B., Su, J., Wen, Z., Jha, D. N., Li, Y., Guan, Y., Puthal, D., James, P., Yang, R., Zomaya, A. Y., et al. (2021) · ACM Computing Surveys · doi:10.1145/3398020 · Cited by 132
Distributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids
Chung, H.-M., Maharjan, S., Zhang, Y., & Eliassen, F. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3007167 · Cited by 120
Coordinated energy management for a cluster of buildings through deep reinforcement learning
Pinto, G., Piscitelli, M. S., Vázquez-Canteli, J. R., Nagy, Z., & Capozzoli, A. (2021) · Energy · doi:10.1016/j.energy.2021.120725 · Cited by 108
Charbonnier, F., Morstyn, T., & McCulloch, M. D. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.118825 · Cited by 78
“Due to the heterogeneity of users and behaviours needing different parameters and models, the large-scale use of model-based controllers is cumbersome [19].”
BackgroundEkanayaka Gunasinghalge, L. U. G., Alazab, A., & Talukder, M. A. (2025) · Energy Informatics · doi:10.1186/s42162-025-00592-8 · Cited by 26
“Wei et al. [20] pioneered Deep Q-Network applications for HVAC control, while Ruelens et al. [17] advanced the field with Proximal Policy Optimization approaches for demand response applications.”
Metadata
- Authors
- Ruelens, F., Claessens, B. J., Vandael, S., De Schutter, B., Babuska, R., & Belmans, R.
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2017-09
- Volume
- 8
- Issue
- 5
- Pages
- 2149-2159
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2277948250
- Cited by
- 333
- References
- 48
- FWCI
- 10.82
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.