The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning
Open accessAbstract
Data center (DC) plays an important role to support services, such as e-commerce and cloud computing. The resulting energy consumption from this growing market has drawn significant attention, and noticeably almost half of the energy cost is used to cool the DC to a particular temperature. It is thus an critical operational challenge to curb the cooling energy cost without sacrificing the thermal safety of a DC. The existing solutions typically follow a two-step approach, in which the system is first modeled based on expert knowledge and, thus, the operational actions are determined with heuristics and/or best practices. These approaches are often hard to generalize and might result in suboptimal performances due to intrinsic model errors for large-scale systems. In this paper, we propose optimizing the DC cooling control via the emerging deep reinforcement learning (DRL) framework. Compared to the existing approaches, our solution lends itself an end-to-end cooling control algorithm (CCA) via an off-policy offline version of the deep deterministic policy gradient (DDPG) algorithm, in which an evaluation network is trained to predict the DC energy cost along with resulting cooling effects, and a policy network is trained to gauge optimized control settings. Moreover, we introduce a de-underestimation (DUE) validation mechanism for the critic network to reduce the potential underestimation of the risk caused by neural approximation. Our proposed algorithm is evaluated on an EnergyPlus simulation platform and on a real data trace collected from the National Super Computing Centre (NSCC) of Singapore. The resulting numerical results show that the proposed CCA can achieve up to 11% cooling cost reduction on the simulation platform compared with a manually configured baseline control algorithm. In the trace-based study of conservative nature, the proposed algorithm can achieve about 15% cooling energy savings on the NSCC data trace. Our pioneering approach can shed new light on the application of DRL to optimize and automate DC operations and management, potentially revolutionizing digital infrastructure management with intelligence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
EnergyPlus: creating a new-generation building energy simulation program
Crawley, D. B., Lawrie, L. K., Winkelmann, F. C., Buhl, W. F., Huang, Y. J., Pedersen, C. O., Strand, R. K., Liesen, R. J., Fisher, D. E., Witte, M. J., et al. (2001) · Energy and Buildings · doi:10.1016/s0378-7788(00)00114-6 · Cited by 2,822
Data Center Energy Consumption Modeling: A Survey
Dayarathna, M., Wen, Y., & Fan, R. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2015.2481183 · Cited by 1,165
Self-citationA Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients
Grondman, I., Busoniu, L., Lopes, G. A. D., & Babuska, R. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2012.2218595 · Cited by 1,042
Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822 · cited 1×
“propose an off-policy RL method to solve nonlinear nonzero-sum games; in [37], Deng et al.”
MethodPolicy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems
Liu, D., & Wei, Q. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2281663 · Cited by 753
Liu, D., Yang, X., Wang, D., & Wei, Q. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2417170 · Cited by 440
Off-Policy Reinforcement Learning for $ H_\infty $ Control Design
Luo, B., Wu, H.-N., & Huang, T. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2319577 · Cited by 362
Reducing energy costs and peak electrical demand through optimal control of building thermal storage
Braun, J. E. (1990) · Cited by 350
Global optimization of absorption chiller system by genetic algorithm and neural network
Chow, T. T., Zhang, G. Q., Lin, Z., & Song, C. (2002) · Energy and Buildings · doi:10.1016/s0378-7788(01)00085-8 · Cited by 336
A review of air conditioning energy performance in data centers
Ni, J., & Bai, X. (2017) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2016.09.050 · Cited by 290
Language Understanding for Text-based Games using Deep Reinforcement Learning
Narasimhan, K., Kulkarni, T., & Barzilay, R. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1001 · Cited by 285
Song, R., Lewis, F. L., & Wei, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582849 · Cited by 233
Optimized Assistive Human–Robot Interaction Using Reinforcement Learning
Modares, H., Ranatunga, I., Lewis, F. L., & Popa, D. O. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2412554 · Cited by 220
Minimizing data center cooling and server power costs
Pakbaznia, E., & Pedram, M. (2009) · Proceedings of the 2009 ACM/IEEE international symposium on Low power electronics and design · doi:10.1145/1594233.1594268 · Cited by 218
HVAC system optimization—in-building section
Lu, L., Cai, W., Xie, L., Li, S., & Soh, Y. C. (2005) · Energy and Buildings · doi:10.1016/j.enbuild.2003.12.007 · Cited by 202
Lo, C.-C., Tsai, S.-H. L., & Lin, B. (2016) · Energies · doi:10.3390/en9020113 · Cited by 151
05/01238 HVAC system optimization-in-building section
(2005) · Fuel and Energy Abstracts · doi:10.1016/s0140-6701(05)81243-3 · Cited by 147
Biomimetic Hybrid Feedback Feedforward Neural-Network Learning Control
Pan, Y., & Yu, H. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2527501 · Cited by 130
Evolving deep unsupervised convolutional networks for vision-based reinforcement learning
Koutník, J., Schmidhuber, J., & Gomez, F. J. (2014) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2576768.2598358 · Cited by 115
Sun, J., & Reddy, A. (2004) · Building and Environment · doi:10.1016/j.buildenv.2004.08.011 · Cited by 113
Towards Joint Optimization Over ICT and Cooling Systems in Data Centre: A Survey
Zhang, W., Wen, Y., Wong, Y. W., Toh, K. C., & Chen, C.-H. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2016.2545109 · Cited by 100
Self-citationFouladi, K., Wemhoff, A. P., Silva-Llanca, L., Abbasi, K., & Ortega, A. (2017) · Applied Thermal Engineering · doi:10.1016/j.applthermaleng.2017.06.057 · Cited by 52
Learning-based power prediction for data centre operations via deep neural networks
Li, Y., Hu, H., Wen, Y., & Zhang, J. (2016) · doi:10.1145/2940679.2940685 · Cited by 37
Self-citationPowerNetS: Coordinating Data Center Network With Servers and Cooling for Power Optimization
Zheng, K., Zheng, W., Li, L., & Wang, X. (2017) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2017.2711567 · Cited by 21
Cited by
A review of green artificial intelligence: Towards a more sustainable future
Bolón-Canedo, V., Morán-Fernández, L., Cancela, B., & Alonso-Betanzos, A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128096 · Cited by 486
Zhang, Z., Chong, A., Pan, Y., Zhang, C., & Lam, K. P. (2019) · Energy and Buildings · doi:10.1016/j.enbuild.2019.07.029 · Cited by 339
Metadata
- Authors
- Li, Y., Wen, Y., Tao, D., & Guan, K.
- Venue
- IEEE Transactions on Cybernetics
- Published
- 2020-05
- Volume
- 50
- Issue
- 5
- Pages
- 2002-2013
- ISSN
- 2168-2267 (ISSN-L)
- 2168-2275
- OpenAlex
- W2964229770
- Cited by
- 274
- References
- 35
- FWCI
- 27.18
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.