Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Abstract Whole building energy model (BEM) is a physics-based modeling method for building energy simulation. It has been widely used in the building industry for code compliance, building design optimization, retrofit analysis, and other uses. Recent research also indicates its strong potential for the control of heating, ventilation and air-conditioning (HVAC) systems. However, its high-order nature and slow computational speed limit its practical application in real-time HVAC optimal control. Therefore, this study proposes a practical control framework (named BEM-DRL) that is based on deep reinforcement learning. The framework is implemented and assessed in a novel radiant heating system in an existing office building as a case study. The complete implementation process is presented in this study, including: building energy modeling for the novel heating system, multi-objective BEM calibration using the Bayesian method and the Genetic Algorithm, deep reinforcement learning training and simulation results evaluation, and control deployment. By analyzing the real-life control deployment data, it is found that BEM-DRL achieves 16.7% heating demand reduction with more than 95% probability compared to the old rule-based control. However, the framework still faces the practical challenges including building energy modeling of novel HVAC systems and multi-objective model calibration. Systematic study is also needed for the design of deep reinforcement learning training to provide a guideline for practitioners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Factorial Sampling Plans for Preliminary Computational Experiments
Morris, M. D. (1991) · Technometrics · doi:10.1080/00401706.1991.10484804 · Cited by 4,943
Thermal comfort: Analysis and applications in environmental engineering
(1972) · Applied Ergonomics · doi:10.1016/s0003-6870(72)80074-7 · Cited by 4,451
2014 International Joint Conference on Neural Networks (IJCNN)
(2014) · doi:10.1109/ijcnn32435.2014 · Cited by 685
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582
Deep Reinforcement Learning for Building HVAC Control
Wei, T., Wang, Y., & Zhu, Q. (2017) · Proceedings of the 54th Annual Design Automation Conference 2017 · doi:10.1145/3061639.3062224 · Cited by 478
Optimal control of HVAC and window systems for natural ventilation through reinforcement learning
Chen, Y., Norford, L. K., Samuelson, H. W., & Malkawi, A. (2018) · Energy and Buildings · doi:10.1016/j.enbuild.2018.03.051 · Cited by 289
Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning
Li, Y., Wen, Y., Tao, D., & Guan, K. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2927410 · Cited by 274
Ascione, F., Bianco, N., De Stasio, C., Mauro, G. M., & Vanoli, G. P. (2016) · Energy and Buildings · doi:10.1016/j.enbuild.2015.11.033 · Cited by 235
Stochastic Model Predictive Control for Building HVAC Systems: Complexity and Conservatism
Ma, Y., Matusko, J., & Borrelli, F. (2015) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2014.2313736 · Cited by 233
De Coninck, R., & Helsen, L. (2016) · Energy and Buildings · doi:10.1016/j.enbuild.2015.11.014 · Cited by 220
Reinforcement learning for energy conservation and comfort in buildings
Dalamagkidis, K., Kolokotsa, D. D., Kalaitzakis, K. C., & Stavrakakis, G. S. (2006) · Building and Environment · doi:10.1016/j.buildenv.2006.07.010 · Cited by 203
Aftab, M., Chen, C., Chau, C.-K., & Rahwan, T. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.07.077 · Cited by 201
Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities
Vázquez-Canteli, J. R., Ulyanin, S., Kämpf, J., & Nagy, Z. (2019) · Sustainable Cities and Society · doi:10.1016/j.scs.2018.11.021 · Cited by 195
Reinforcement learning for optimal control of low exergy buildings
Yang, L., Nagy, Z., Goffin, P., & Schlueter, A. (2015) · Applied Energy · doi:10.1016/j.apenergy.2015.07.050 · Cited by 182
Huang, H., Chen, L., & Hu, E. (2015) · Building and Environment · doi:10.1016/j.buildenv.2015.01.037 · Cited by 181
Guidelines for the Bayesian calibration of building energy models
Chong, A., & Menberg, K. (2018) · Energy and Buildings · doi:10.1016/j.enbuild.2018.06.028 · Cited by 174
Self-citationModel-predictive control of mixed-mode buildings with rule extraction
May-Ostendorp, P., Henze, G. P., Corbin, C. D., Rajagopalan, B., & Felsmann, C. (2011) · Building and Environment · doi:10.1016/j.buildenv.2010.08.004 · Cited by 171
Trial results from a model predictive control and optimisation system for commercial building HVAC
West, S. R., Ward, J. K., & Wall, J. (2014) · Energy and Buildings · doi:10.1016/j.enbuild.2013.12.037 · Cited by 170
Wang, Y., Velswamy, K., & Huang, B. (2017) · Processes · doi:10.3390/pr5030046 · Cited by 157
Experimental analysis of data-driven control for a building heating system
Costanzo, G. T., Iacovella, S., Ruelens, F., Leurs, T., & Claessens, B. (2016) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2016.02.002 · Cited by 152
Bayesian calibration of building energy models with large datasets
Chong, A., Lam, K. P., Pozzi, M., & Yang, J. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.08.069 · Cited by 147
Self-citationApplication of economic MPC to the energy and demand minimization of a commercial building
Ma, J., Qin, S. J., & Salsbury, T. I. (2014) · Journal of Process Control · doi:10.1016/j.jprocont.2014.06.011 · Cited by 144
Zhang, Z., & Lam, K. P. (2018) · ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys) · doi:10.1145/3276774.3276775 · Cited by 131
Self-citationOptimal exergy control of building HVAC system
Razmara, M., Maasoumy, M., Shahbakhti, M., & Robinett, R. D. (2015) · Applied Energy · doi:10.1016/j.apenergy.2015.07.051 · Cited by 128
Liu, S., & Henze, G. P. (2005) · Energy and Buildings · doi:10.1016/j.enbuild.2005.06.001 · Cited by 124
Li, S., Joe, J., Hu, J., & Karava, P. (2015) · Solar Energy · doi:10.1016/j.solener.2014.11.024 · Cited by 121
Kwak, Y., Huh, J.-H., & Jang, C.-Y. (2015) · Applied Energy · doi:10.1016/j.apenergy.2015.05.096 · Cited by 110
Chen, X., Wang, Q., & Srebric, J. (2015) · Applied Energy · doi:10.1016/j.apenergy.2015.11.065 · Cited by 105
Using reinforcement learning to optimize occupant comfort and energy usage in HVAC systems
Fazenda, P., Veeramachaneni, K. K., Lima, P. U., & O’Reilly, U.-M. (2014) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-140288 · Cited by 96
MPC control for improving energy efficiency of a building air handler for multi-zone VAVs
Liang, W., Quinte, R., Jia, X., & Sun, J. (2015) · Building and Environment · doi:10.1016/j.buildenv.2015.04.033 · Cited by 86
Energy Efficient Building HVAC Control Algorithm with Real-time Occupancy Prediction
Shi, J., Yu, N., & Yao, W. (2017) · Energy Procedia · doi:10.1016/j.egypro.2017.03.028 · Cited by 85
Continuous-time Bayesian calibration of energy models using BIM and energy data
Chong, A., Xu, W., Chao, S., & Ngo, N.-T. (2019) · Energy and Buildings · doi:10.1016/j.enbuild.2019.04.017 · Cited by 79
Self-citationZhao, J., Lam, K. P., Ydstie, B. E., & Karaguzel, O. T. (2014) · Journal of Building Performance Simulation · doi:10.1080/19401493.2014.891656 · Cited by 76
Self-citationRuelens, F., Iacovella, S., Claessens, B., & Belmans, R. (2015) · Energies · doi:10.3390/en8088300 · Cited by 74
Deep Reinforcement Learning for Optimal Control of Space HeatingPreprint
Nagy, A., Kazmi, H., Cheaib, F., & Driesen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.03777 · Cited by 74
The Neurothermostat: Predictive Optimal Control of Residential Heating Systems
Mozer, M. C., Vidmar, L., & Dodier, R. H. (1996) · Cited by 73
Field tests of an adaptive, model-predictive heating controller for residential buildings
Lindelöf, D., Afshari, H., Alisafaee, M., Biswas, J., Caban, M., Mocellin, X., & Viaene, J. (2015) · Energy and Buildings · doi:10.1016/j.enbuild.2015.04.029 · Cited by 71
Model-based energy efficient control applied to an office building
Váňa, Z., Cigler, J., Široký, J., Žáčeková, E., & Ferkl, L. (2014) · Journal of Process Control · doi:10.1016/j.jprocont.2014.01.016 · Cited by 69
Liu, S., & Henze, G. P. (2007) · Journal of Solar Energy Engineering · doi:10.1115/1.2710491 · Cited by 56
宮良, 明., & 香川, 澄. (2010) · 冷凍 · Cited by 55
Implementation of cooperative Fuzzy model predictive control for an energy-efficient office building
Killian, M., & Kozek, M. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.11.021 · Cited by 53
Yu, Z., & Dexter, A. L. (2010) · Control Engineering Practice · doi:10.1016/j.conengprac.2010.01.018 · Cited by 52
Vázquez-Canteli, J., Kämpf, J., & Nagy, Z. (2017) · Energy Procedia · doi:10.1016/j.egypro.2017.07.429 · Cited by 42
A learning agent for heat-pump thermostat control
Urieli, D., & Stone, P. (2013) · doi:10.5555/2484920.2485092 · Cited by 36
Prioritised objectives for model predictive control of building heating systems
O’Dwyer, E., Tommasi, L. D., Kouramas, K. I., Cychowski, M. T., & Lightbody, G. (2017) · Control Engineering Practice · doi:10.1016/j.conengprac.2017.03.018 · Cited by 29
Model-based predictive control for building energy management: Part II – Experimental validations
Yu, N., Salakij, S., Chavez-Martinez, R., Paolucci, S., Sen, M., & Antsaklis, P. J. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.04.027 · Cited by 27
Model Predictive Control for Hydronic Heating Systems in Residential Buildings
Fielsch, S., Grunert, T., Stursberg, M., & Kummert, A. (2017) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2017.08.817 · Cited by 11
Cited by
Reinforcement learning for building controls: The opportunities and challenges
Wang, Z., & Hong, T. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115036 · Cited by 577
A Review of Deep Reinforcement Learning for Smart Building Energy Management
Yu, L., Qin, S., Zhang, M., Shen, C., Jiang, T., & Guan, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3078462 · Cited by 476
Applications of reinforcement learning in energy systems
Perera, A. T. D., & Kamalaruban, P. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.110618 · Cited by 420
A practical guide to multi-objective reinforcement learning and planning
Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., et al. (2022) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-022-09552-y · Cited by 400
“…systems [Lacerda, 2017], power systems [Deng and Liu, 2018, Deng et al., 2020, Wang et al., 2019, Mello et al., 2020], building management [Zhang et al., 2019], traffic management [Jin and Ma, 2019], manufacturing [Govindaiah and Petty, 2019, Lepenioti et al., 2020, Dornheim and Link,…”
BackgroundMulti-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings
Yu, L., Sun, Y., Xu, Z., Shen, C., Yue, D., Jiang, T., & Guan, X. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3011739 · Cited by 353
Metadata
- Authors
- Zhang, Z., Chong, A., Pan, Y., Zhang, C., & Lam, K. P.
- Venue
- Energy and Buildings
- Publisher
- Elsevier BV
- Published
- 2019-09
- Volume
- 199
- Pages
- 472-490
- ISSN
- 0378-7788 (ISSN-L)
- OpenAlex
- W2957897705
- Cited by
- 339
- References
- 54
- FWCI
- 19.72
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.