Deep reinforcement learning to optimise indoor temperature control and heating energy consumption in buildings
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
In this work, Deep Reinforcement Learning (DRL) is implemented to control the supply water temperature setpoint to terminal units of a heating system. The experiment was carried out for an office building in an integrated simulation environment. A sensitivity analysis is carried out on relevant hyperparameters to identify their optimal configuration. Moreover, two sets of input variables were considered for assessing their impact on the adaptability capabilities of the DRL controller. In this context a static and dynamic deployment of the DRL controller is performed. The trained control agent is tested for four different scenarios to determine its adaptability to the variation of forcing variables such as weather conditions, occupant presence patterns and different indoor temperature setpoint requirements. The performance of the agent is evaluated against a reference controller that implements a combination of rule-based and climatic-based logics. As a result, when the set of variables are adequately selected a heating energy saving ranging between 5 and 12% is obtained with an enhanced indoor temperature control with both static and dynamic deployment. Eventually the study proves that if the set of input variables are not carefully selected a dynamic deployment is strictly required for obtaining good performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Reinforcement Learning: An Introduction
(1998) · Kybernetes · doi:10.1108/k.1998.27.9.1093.3 · Cited by 1,472
Theory and applications of HVAC control systems – A review of model predictive control (MPC)
Afram, A., & Janabi-Sharifi, F. (2014) · Building and Environment · doi:10.1016/j.buildenv.2013.11.016 · Cited by 1,310
Reinforcement learning for demand response: A review of algorithms and modeling techniques
Vázquez-Canteli, J. R., & Nagy, Z. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.11.002 · Cited by 883
EnergyPlus エネルギー・シミュレーション・プログラム
Crawley, D. B., Lawrie, L. K., PEDERSEN, C. O., & Winkelmann, F. C. (2000) · Cited by 725
Afram, A., Janabi-Sharifi, F., Fung, A. S., & Raahemifar, K. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.02.012 · Cited by 655
Serale, G., Fiorentini, M., Capozzoli, A., Bernardini, D., & Bemporad, A. (2018) · Energies · doi:10.3390/en11030631 · Cited by 637
Self-citationDeep Reinforcement Learning for Building HVAC Control
Wei, T., Wang, Y., & Zhu, Q. (2017) · Proceedings of the 54th Annual Design Automation Conference 2017 · doi:10.1145/3061639.3062224 · Cited by 478
Building modeling as a crucial part for building predictive control
Prívara, S., Cigler, J., Váňa, Z., Oldewurtel, F., Sagerschnig, C., & Žáčeková, E. (2013) · Energy and Buildings · doi:10.1016/j.enbuild.2012.10.024 · Cited by 348
Zhang, Z., Chong, A., Pan, Y., Zhang, C., & Lam, K. P. (2019) · Energy and Buildings · doi:10.1016/j.enbuild.2019.07.029 · Cited by 339
Deep Reinforcement Learning in Large Discrete Action Spaces
Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., & Coppin, B. (2015) · arXiv · doi:10.48550/arxiv.1512.07679 · Cited by 290
Economic Model Predictive Control for building climate control in a Smart Grid
Halvgaard, R., Poulsen, N. K., Madsen, H., & Jorgensen, J. B. (2012) · 2012 IEEE PES Innovative Smart Grid Technologies (ISGT) · doi:10.1109/isgt.2012.6175631 · Cited by 289
Data science for building energy management: A review
Molina-Solana, M., Ros, M., Ruiz, M. D., Gómez-Romero, J., & Martin-Bautista, M. J. (2017) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2016.11.132 · Cited by 287
Fan, C., Xiao, F., & Yan, C. (2014) · Automation in Construction · doi:10.1016/j.autcon.2014.12.006 · Cited by 218
A comparative review of heating systems in EU countries, based on efficiency and fuel cost
Martinopoulos, G., Papakostas, K. T., & Papadopoulos, A. M. (2018) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2018.03.060 · Cited by 203
Valladares, W., Galindo, M., Gutiérrez, J., Wu, W.-C., Liao, K.-K., Liao, J.-C., Lu, K.-C., & Wang, C.-C. (2019) · Building and Environment · doi:10.1016/j.buildenv.2019.03.038 · Cited by 200
Review of Control Techniques for HVAC Systems—Nonlinearity Approaches Based on Fuzzy Cognitive Maps
Behrooz, F., Mariun, N., Marhaban, M., Mohd Radzi, M., & Ramli, A. (2018) · Energies · doi:10.3390/en11030495 · Cited by 196
Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities
Vázquez-Canteli, J. R., Ulyanin, S., Kämpf, J., & Nagy, Z. (2019) · Sustainable Cities and Society · doi:10.1016/j.scs.2018.11.021 · Cited by 195
Zou, Z., Yu, X., & Ergan, S. (2020) · Building and Environment · doi:10.1016/j.buildenv.2019.106535 · Cited by 188
A review of reinforcement learning methodologies for controlling occupant comfort in buildings
Han, M., May, R., Zhang, X., Wang, X., Pan, S., Yan, D., Jin, Y., & Xu, L. (2019) · Sustainable Cities and Society · doi:10.1016/j.scs.2019.101748 · Cited by 162
Wang, Y., Velswamy, K., & Huang, B. (2017) · Processes · doi:10.3390/pr5030046 · Cited by 157
Capozzoli, A., Piscitelli, M. S., Brandi, S., Grassi, D., & Chicco, G. (2018) · Energy · doi:10.1016/j.energy.2018.05.127 · Cited by 153
Self-citationExperimental analysis of data-driven control for a building heating system
Costanzo, G. T., Iacovella, S., Ruelens, F., Leurs, T., & Claessens, B. (2016) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2016.02.002 · Cited by 152
A SURVEY OF CONTROL TECHNOLOGIES IN THE BUILDING AUTOMATION INDUSTRY
Salsbury, T. I. (2005) · IFAC Proceedings Volumes · doi:10.3182/20050703-6-cz-1902.01397 · Cited by 134
Explorations in efficient reinforcement learning
Wiering, M. (1999) · UvA-DARE (University of Amsterdam) · Cited by 108
Using reinforcement learning to optimize occupant comfort and energy usage in HVAC systems
Fazenda, P., Veeramachaneni, K. K., Lima, P. U., & O’Reilly, U.-M. (2014) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-140288 · Cited by 96
Extracting knowledge from building-related data — A data mining framework
Yu, Z. J., Fung, B. C. M., & Haghighat, F. (2013) · Building Simulation · doi:10.1007/s12273-013-0117-8 · Cited by 96
Control Strategies for Building Energy Systems to Unlock Demand Side Flexibility – A Review
Clauß, J., Finck, C., Vogler-Finck, P. J. C., & Beagon, P. (2017) · Building Simulation Conference proceedings · doi:10.26868/25222708.2017.462 · Cited by 91
Naidu, D. S., & Rieger, C. G. (2011) · HVAC&R Research · doi:10.1080/10789669.2011.540942 · Cited by 87
Building temperature regulation in a multi-zone HVAC system using distributed adaptive control
Lymperopoulos, G., & Ioannou, P. (2020) · Energy and Buildings · doi:10.1016/j.enbuild.2020.109825 · Cited by 73
Modeling and identification of a large multi-zone office building
Privara, S., Vana, Z., Gyalistras, D., Cigler, J., Sagerschnig, C., Morari, M., & Ferkl, L. (2011) · 2011 IEEE International Conference on Control Applications (CCA) · doi:10.1109/cca.2011.6044402 · Cited by 70
Piscitelli, M. S., Brandi, S., & Capozzoli, A. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.113727 · Cited by 67
Self-citationMining typical load profiles in buildings to support energy management in the smart city context
Capozzoli, A., Piscitelli, M. S., & Brandi, S. (2017) · Energy Procedia · doi:10.1016/j.egypro.2017.09.545 · Cited by 59
Self-citationAdvanced control strategies for HVAC&R systems—An overview: Part II: Soft and fusion control
Naidu, D. S., & Rieger, C. G. (2011) · Science and Technology for the Built Environment · doi:10.1080/10789669.2011.555650 · Cited by 56
Mady, A. E.-D., Provan, G., Ryan, C., & Brown, K. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7802 · Cited by 54
Fiorentini, M., Cooper, P. W., Ma, Z., & Robinson, D. (2015) · Energy Procedia · doi:10.1016/j.egypro.2015.12.192 · Cited by 47
Vázquez-Canteli, J., Kämpf, J., & Nagy, Z. (2017) · Energy Procedia · doi:10.1016/j.egypro.2017.07.429 · Cited by 42
Baldi, S., Zhang, F., Quang, T. L., Endel, P., & Holub, O. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.113478 · Cited by 26
Cited by
An overview of machine learning applications for smart buildings
Alanne, K., & Sierla, S. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2021.103445 · Cited by 280
Reinforcement learning for HVAC control in intelligent buildings: A technical and conceptual review
Al Sayed, K., Boodi, A., Sadeghian Broujeny, R., & Beddiar, K. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.110085 · Cited by 140
Evaluation of advanced control strategies for building energy systems
Stoffel, P., Maier, L., Kümpel, A., Schreiber, T., & Müller, D. (2023) · Energy and Buildings · doi:10.1016/j.enbuild.2022.112709 · Cited by 126
“Another study on using RL for BES energy management applications was published by Brandi et al. ([72]).”
Coordinated energy management for a cluster of buildings through deep reinforcement learning
Pinto, G., Piscitelli, M. S., Vázquez-Canteli, J. R., Nagy, Z., & Capozzoli, A. (2021) · Energy · doi:10.1016/j.energy.2021.120725 · Cited by 108
Self-citationAdesanya, M. A., Obasekore, H., Rabiu, A., Na, W.-H., Ogunlowo, Q. O., Akpenpuun, T. D., Kim, M.-H., Kim, H.-T., Kang, B.-Y., & Lee, H.-W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124126 · Cited by 72
Metadata
- Authors
- Brandi, S., Piscitelli, M. S., Martellacci, M., & Capozzoli, A.
- Venue
- Energy and Buildings
- Publisher
- Elsevier BV
- Published
- 2020-10
- Volume
- 224
- Pages
- 110225
- ISSN
- 0378-7788 (ISSN-L)
- OpenAlex
- W3034748593
- Cited by
- 258
- References
- 42
- FWCI
- 15.23
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.