The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning
Open accessAbstract
Driven by recent advances in batch Reinforcement Learning (RL), this paper contributes to the application of batch RL to demand response. In contrast to conventional model-based approaches, batch RL techniques do not require a system identification step, making them more suitable for a large-scale implementation. This paper extends fitted Q-iteration, a standard batch RL technique, to the situation when a forecast of the exogenous data is provided. In general, batch RL techniques do not rely on expert knowledge about the system dynamics or the solution. However, if some expert knowledge is provided, it can be incorporated by using the proposed policy adjustment method. Finally, we tackle the challenge of finding an open-loop schedule required to participate in the day-ahead market. We propose a model-free Monte Carlo method that uses a metric based on the state-action value function or Q-function and we illustrate this method by finding the day-ahead schedule of a heat-pump thermostat. Our experiments show that batch RL techniques provide a valuable alternative to model-based controllers and that they can be used to construct both closed-loop and open-loop policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
On-line Q-learning Using Connectionist Systems
Rummery, G. A. (1994) · Cited by 1,477
Tree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1088690 · Cited by 1,352
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Reinforcement Learning and Dynamic Programming Using Function Approximators
Busoniu, L., Babuska, R., De Schutter, B., & Ernst, D. (2017) · CRC Press eBooks · doi:10.1201/9781439821091 · Cited by 919
Self-citationvan Hasselt, H. P. (2010) · Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands · Cited by 839
Riedmiller, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564096_32 · Cited by 757
Management and Control of Domestic Smart Grid Technology
Molderink, A., Bakker, V., Bosman, M. G. C., Hurink, J. L., & Smit, G. J. M. (2010) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2010.2055904 · Cited by 624
Feature-based methods for large scale dynamic programming
Tsitsiklis, J. N., & van Roy, B. (1996) · Machine Learning · doi:10.1007/bf00114724 · Cited by 432
GridLAB-D: An open-source power systems modeling and simulation environment
Chassin, D. P., Schneider, K., & Gerkensmeyer, C. (2008) · 2008 IEEE/PES Transmission and Distribution Conference and Exposition · doi:10.1109/tdc.2008.4517260 · Cited by 420
Relational Reinforcement Learning
Džeroski, S., De Raedt, L., & Driessens, K. (2001) · Machine Learning · doi:10.1023/a:1007694015589 · Cited by 371
Lange, S., Gabel, T., & Riedmiller, M. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_2 · Cited by 370
Kernel-Based Reinforcement Learning
(2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_100235 · Cited by 367
Annual Energy Outlook 2011 with Projections to 2035
U.S. Energy Information Administration, O. O. I. A. I. E. A. (2011) · doi:10.2172/1019039 · Cited by 355
Deep auto-encoder neural networks in reinforcement learning
Lange, S., & Riedmiller, M. (2010) · The 2010 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596468 · Cited by 329
Variable Resolution Discretization in Optimal Control
Munos, R., & Moore, A. (2002) · Machine Learning · doi:10.1023/a:1017992615625 · Cited by 321
Computationally Feasible Bounds for Partially Observed Markov Decision Processes
Lovejoy, W. S. (1991) · Operations Research · doi:10.1287/opre.39.1.162 · Cited by 307
Koch, S., Mathieu, J. L., & Callaway, D. S. (2011) · Cited by 285
Residential Demand Response Using Reinforcement Learning
O'Neill, D., Levorato, M., Goldsmith, A., & Mitra, U. (2010) · 2010 First IEEE International Conference on Smart Grid Communications · doi:10.1109/smartgrid.2010.5622078 · Cited by 281
Optimal Demand Response Using Device-Based Reinforcement Learning
Wen, Z., O'Neill, D., & Maei, H. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2396993 · Cited by 266
Experience Replay for Real-Time Reinforcement Learning Control
Adam, S., Busoniu, L., & Babuska, R. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2106494 · Cited by 255
Self-citationKernel-Based Reinforcement Learning
Ormoneit, D., & Sen, Ś. (2002) · Machine Learning · doi:10.1023/a:1017928328829 · Cited by 255
A Scalable Three-Step Approach for Demand Side Management of Plug-in Hybrid Vehicles
Vandael, S., Claessens, B., Hommelberg, M., Holvoet, T., & Deconinck, G. (2013) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2012.2213847 · Cited by 240
Self-citationReinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem
Ernst, D., Glavic, M., Capitanescu, F., & Wehenkel, L. (2009) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.2007630 · Cited by 214
Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market
Vandael, S., Claessens, B., Ernst, D., Holvoet, T., & Deconinck, G. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2015.2393059 · Cited by 208
Self-citationPerformance Bounds in $L_p$‐norm for Approximate Value Iteration
Munos, R. (2007) · SIAM Journal on Control and Optimization · doi:10.1137/040614384 · Cited by 184
State Estimation and Control of Heterogeneous Thermostatically Controlled Loads for Load Following
Mathieu, J. L., & Callaway, D. S. (2012) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2012.545 · Cited by 121
Evaluation of Reinforcement Learning Control for Thermal Energy Storage Systems
Henze, G. P., & Schoenmann, J. (2003) · Science and Technology for the Built Environment · doi:10.1080/10789669.2003.10391069 · Cited by 107
PowerMatching City, a living lab smart grid demonstration
Bliek, F. W., van den Noort, A., Roossien, B., Kamphuis, R. I. G., de Wit, J., van der Velde, J., & Eijgelaar, M. (2010) · IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT Europe) · doi:10.1109/isgteurope.2010.5638863 · Cited by 102
A Smart Domestic Hot Water Buffer
Vanthournout, K., D’hulst, R., Geysen, D., & Jacobs, G. (2012) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2012.2205591 · Cited by 100
Multiagent reinforcement learning using function approximation
Abul, O., Polat, F., & Alhajj, R. (2000) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/5326.897075 · Cited by 96
Ruelens, F., Claessens, B., Vandael, S., Iacovella, S., Vingerhoets, P., & Belmans, R. J. M. (2014) · Power Systems Computation Conference · doi:10.1109/pscc.2014.7038106 · Cited by 86
Self-citationDupont, B., Vingerhoets, P., Tant, P., Vanthournout, K., Cardinaels, W., Rybel, T. D., Peeters, E., & Belmans, R. J. M. (2012) · IEEE PES Innovative Smart Grid Technologies Europe · doi:10.1109/isgteurope.2012.6465708 · Cited by 67
Self-citationStochastic Control for Smart Grid Users With Flexible Demand
Liang, Y., He, L., Cao, X., & Shen, Z. M. (2013) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2013.2263201 · Cited by 55
Hierarchical Multiagent Reinforcement Learning
Ghavamzadeh, M., & Mahadevan, S. (2004) · Defense Technical Information Center (DTIC) · doi:10.21236/ada440418 · Cited by 54
Approximate Dynamic Programming
Bertsekas, D. P. (2018) · CRC Press eBooks · doi:10.1201/b10384-137 · Cited by 50
Batch mode reinforcement learning based on the synthesis of artificial trajectories
Raphael Fonteneau, D., Murphy, S. A., Wehenkel, L. A., & Ernst, D. (2012) · Annals of Operations Research · doi:10.1007/s10479-012-1248-5 · Cited by 42
Shaping multi-agent systems with gradient reinforcement learning
Buffet, O., Dutech, A., & Charpillet, F. (2007) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-006-9010-5 · Cited by 38
Learning in Multiagent Systems: An Introduction from a Game-Theoretic Perspective
Vidal, J. M. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44826-8_13 · Cited by 26
An Intelligent Battery Controller Using Bias-Corrected Q-learning
Lee, D., & Powell, W. B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8164 · Cited by 22
Multi-agent learning for routing control within an Internet environment
Tillotson, P. R. J., Wu, Q., & Hughes, P. M. (2004) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2004.02.004 · Cited by 18
Multiagent reinforcement learning applied to a chase problem in a continuous world
Tamakoshi, H., & Ishii, S. (2001) · Artificial Life and Robotics · doi:10.1007/bf02481502 · Cited by 15
Cited by
Reinforcement learning for demand response: A review of algorithms and modeling techniques
Vázquez-Canteli, J. R., & Nagy, Z. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.11.002 · Cited by 883
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
“Electricity Systems, especially intelligent electricity networks, have been also built based on RL techniques for power-delivery decisions [176], [177] and active resource management [178].”
BackgroundDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Antonopoulos, I., Robu, V., Couraud, B., Kirli, D., Norbu, S., Kiprakis, A., Flynn, D., Elizondo-Gonzalez, S., & Wattam, S. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.109899 · Cited by 656
Reinforcement learning for building controls: The opportunities and challenges
Wang, Z., & Hong, T. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115036 · Cited by 577
Metadata
- Authors
- Ruelens, F., Claessens, B. J., Vandael, S., De Schutter, B., Babuska, R., & Belmans, R.
- Venue
- IEEE Transactions on Smart Grid
- Published
- 2017-09
- Volume
- 8
- Issue
- 5
- Pages
- 2149-2159
- ISSN
- 1949-3053 (ISSN-L)
- 1949-3061
- OpenAlex
- W2277948250
- Cited by
- 333
- References
- 48
- FWCI
- 10.82
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.