Resource Management with Deep Reinforcement Learning
Abstract
Resource management problems in systems and networking often manifest as difficult online decision making tasks where appropriate solutions depend on understanding the workload and environment. Inspired by recent advances in deep reinforcement learning for AI problems, we consider building systems that learn to manage resources directly from experience. We present DeepRM, an example solution that translates the problem of packing tasks with multiple resource demands into a learning problem. Our initial results show that DeepRM performs comparably to state-of-the-art heuristics, adapts to different conditions, converges quickly, and learns strategies that are sensible in hindsight.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RL combined with deep-learning has been shown as a promising alternative that can be leveraged to tackle various resource management problems in practical settings.
“Recently, RL combined with deep-learning has been shown as a promising alternative [294] that can be leveraged to tackle various resource management problems in practical settings.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [294]
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Monte Carlo sampling methods using Markov chains and their applications.
Hastings, W. K. (1970) · Biometrika · doi:10.1093/biomet/57.1.97 · Cited by 15,195 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., et al. (2010) · Communications of the ACM · doi:10.1145/1721654.1721672 · Cited by 9,464 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
(1999) · Cited by 5,147 · see citing works
Hagan, M., Demuth, H. B., Beale, M. H., & Jes?s, O. D. (2016) · Institutional Repository of the National University Open and Distance UNAD (Universidad Nacional Abierta y a Distancia) · Cited by 5,133 · see citing works
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Dean, J., & Barroso, L. A. (2013) · Communications of the ACM · doi:10.1145/2408776.2408794 · Cited by 1,852 · see citing works
Apache Hadoop YARN: yet another resource negotiator
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., et al. (2013) · Proceedings of the 4th annual Symposium on Cloud Computing · doi:10.1145/2523616.2523633 · Cited by 1,848 · see citing works
Zaharia, M., Borthakur, D., Sarma, J. S., Elmeleegy, K., Shenker, S. J., & Stoica, I. G. (2010) · European Conference on Computer Systems · doi:10.1145/1755913.1755940 · Cited by 1,394 · see citing works
Dominant resource fairness: fair allocation of multiple resource types
Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S. J., & Stoica, I. G. (2011) · doi:10.5555/1972457.1972490 · Cited by 1,120 · see citing works
A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP
Yin, X., Jindal, A., Sekar, V., & Sinopoli, B. (2015) · ACM Special Interest Group on Data Communication · doi:10.1145/2785956.2787486 · Cited by 1,030 · see citing works
(2010) · doi:10.5555/1855711.1855728 · Cited by 970 · see citing works
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., & Goldberg, A. V. (2009) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/1629575.1629601 · Cited by 856 · see citing works
Monahan, G. (1982) · Management Science · doi:10.1287/mnsc.28.1.1 · Cited by 736 · see citing works
Delimitrou, C., & Kozyrakis, C. (2014) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2541940.2541941 · Cited by 704 · see citing works
Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach
Boyan, J. A., & Littman, M. L. (1994) · Cited by 694 · see citing works
Reining in the outliers in map-reduce clusters using Mantri
Ananthanarayanan, G., Kandula, S., Greenberg, A. G., Stoica, I. G., Lu, Y., Saha, B., & Harris, E. F. (2010) · Operating Systems Design and Implementation · doi:10.5555/1924943.1924962 · Cited by 660 · see citing works
Self-citationLearning policies for partially observable environments: Scaling up
Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50052-9 · Cited by 659 · see citing works
Project Adam: building an efficient and scalable deep learning training system
Chilimbi, T. M., Suzue, Y., Apacible, J., & Kalyanaraman, K. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685094 · Cited by 614 · see citing works
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0006 · Cited by 551 · see citing works
Neuro-dynamic programming: an overview
Bertsekas, D. P., & Tsitsiklis, J. N. (2002) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.1995.478953 · Cited by 424 · see citing works
Multi-resource packing for cluster schedulers
Grandl, R., Ananthanarayanan, G., Kandula, S., Rao, S., & Akella, A. (2014) · Proceedings of the 2014 ACM conference on SIGCOMM · doi:10.1145/2619239.2626334 · Cited by 421 · see citing works
Self-citationCS2P: Improving Video Bitrate Selection and Adaptation with Data-Driven Throughput Prediction
Sun, Y., Yin, X., Jiang, J., Sekar, V., Lin, F., Wang, N., Liu, T., & Sinopoli, B. (2016) · Proceedings of the 2016 ACM SIGCOMM Conference · doi:10.1145/2934872.2934898 · Cited by 407 · see citing works
A reinforcement learning approach to job-shop scheduling
Zhang, W., & Dietterich, T. G. (1995) · Cited by 374 · see citing works
Winstein, K., & Balakrishnan, H. (2013) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2486001.2486020 · Cited by 332 · see citing works
PCC: Re-architecting Congestion Control for Consistent High PerformancePreprint
Dong, M., Li, Q., Zarchy, D., Godfrey, B., & Schapira, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.7092 · Cited by 298 · see citing works
Ferguson, A. D., Bodík, P., Kandula, S., Boutin, E., & Fonseca, R. (2012) · European Conference on Computer Systems · doi:10.1145/2168836.2168847 · Cited by 286 · see citing works
Self-citationStochastic forecasts achieve high throughput and low delay over cellular networks
Winstein, K., Sivaraman, A., & Balakrishnan, H. (2013) · Cited by 249 · see citing works
Machine Learning Proceedings 1995
(1995) · Elsevier eBooks · doi:10.1016/c2009-0-27705-1 · Cited by 226 · see citing works
Basis Function Adaptation in Temporal Difference Reinforcement Learning
Menache, I., Mannor, S., & Shimkin, N. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-5732-z · Cited by 184 · see citing works
Self-citationRe-optimizing data-parallel computing
Agarwal, S., Kandula, S., Bruno, N., Wu, M.-C., Stoica, I. G., & Zhou, J. (2012) · Cited by 174 · see citing works
Self-citationJiang, J., Das, R., Ananthanarayanan, G., Chou, P. A., Padmanabhan, V. N., Sekar, V., Dominique, E., Goliszewski, M., Kukoleca, D., Vafin, R., et al. (2016) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2934872.2934907 · Cited by 97 · see citing works
Optimizing Production Manufacturing Using Reinforcement Learning
Mahadevan, S., & Theocharous, G. (1998) · Cited by 64 · see citing works
Flagged first, then by citations
1 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D. T., Wang, P., Liang, Y., & Kim, D. I. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916583 · Cited by 2,111 · see citing works
“The authors in [163] propose a DRL-based solution, DeepRM, by employing policy gradient methods [148] to manage resources in computer systems directly from experience.”
Method6G and Beyond: The Future of Wireless Communications Systems
Akyildiz, I. F., Kak, A., & Nie, S. (2020) · IEEE Access · doi:10.1109/access.2020.3010896 · Cited by 1,419 · see citing works
“Reinforcement Learning: Reinforcement learning typically finds use in problems associated with resource management [97], [98].”
BackgroundNeural Adaptive Video Streaming with Pensieve
Mao, H., Netravali, R., & Alizadeh, M. (2017) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3098822.3098843 · Cited by 1,378 · see citing works
“Pensieve uses modern reinforcement learning (RL) techniques [27, 29, 42] to learn a control policy for bitrate adaptation purely through experience.”
MethodSelf-citationOptimizing Federated Learning on Non-IID Data with Reinforcement Learning
Wang, H., Kaplan, Z., Niu, D., & Li, B. (2020) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom41043.2020.9155494 · Cited by 1,054 · see citing works
“The recent success of RL [12, 13] comes from the capability that RL agents learn from the interaction with a dynamic environment.”
BackgroundBoutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
“Recently, RL combined with deep-learning has been shown as a promising alternative [294] that can be leveraged to tackle various resourcemanagement problems in practical settings.”
BackgroundMethodDeep Reinforcement Learning Based Resource Allocation for V2V Communications
Ye, H., Li, G. Y., & Juang, B.-H. F. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2897134 · Cited by 870 · see citing works
Layer-Wise Relevance Propagation: An Overview
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_10 · Cited by 825 · see citing works
Deep Learning for Intelligent Wireless Networks: A Comprehensive Survey
Mao, Q., Hu, F., & Hao, Q. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2846401 · Cited by 789 · see citing works
“[90] proposed to use reinforcement learning in”
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Learning scheduling algorithms for data processing clusters
Mao, H., Schwarzkopf, M., Venkatakrishnan, S. B., Meng, Z., & Alizadeh, M. (2018) · Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication · doi:10.1145/3341302.3342080 · Cited by 899 · see citing works
“For example, if the objective is to minimize average JCT, Decima penalizes the agent −τ × J at each time step, where τ is the absolute time (in seconds) since last action and J is the number of jobs in the system [28].”
BackgroundMethodInfluentialSelf-citationAntonopoulos, I., Robu, V., Couraud, B., Kirli, D., Norbu, S., Kiprakis, A., Flynn, D., Elizondo-Gonzalez, S., & Wattam, S. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.109899 · Cited by 656 · see citing works
“The RL framework has been applied to a number of domains with the most important being robotics [102,103], resource management in computer clusters [104], playing video games from pixel input [105], automated ML frameworks [106].”
Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G
Vaezi, M., Azari, A., Khosravirad, S. R., Shirvanimoghaddam, M., Azari, M. M., Chasaki, D., & Popovski, P. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3151028 · Cited by 654 · see citing works
Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning
Liang, L., Ye, H., & Li, G. Y. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2933962 · Cited by 559 · see citing works
“In particular, recent success of deep RL in humanlevel video game play [7] and AlphaGo [8] has sparked a flurry of interest in applying RL techniques to solve problems from a wide variety of areas and remarkable progress has been made ever since [9], [10], [11].”
BackgroundApplications of artificial intelligence for disaster management
Sun, W., Bocchini, P., & Davison, B. D. (2020) · Natural Hazards · doi:10.1007/s11069-020-04124-3 · Cited by 491 · see citing works
Machine Learning for Networking: Workflow, Advances and Opportunities
Wang, M., Cui, Y., Wang, X., Xiao, S., & Jiang, J. (2017) · IEEE Network · doi:10.1109/mnet.2017.1700200 · Cited by 448 · see citing works
“[13] H.”
BackgroundInfluentialSemisupervised Deep Reinforcement Learning in Support of IoT and Smart City Services
Mohammadi, M., Al‐Fuqaha, A., Guizani, M., & Oh, J.-S. (2017) · IEEE Internet of Things Journal · doi:10.1109/jiot.2017.2712560 · Cited by 435 · see citing works
“[22] formulated the problem of job scheduling with multiple resource demands as a DRL process.”
BackgroundMestres, A., Rodriguez-Natal, A., Carner, J., Barlet‐Ros, P., Alarcón, E., Solé, M., Muntés-Mulero, V., Meyer, D., Barkai, S., Hibbett, M. J., et al. (2017) · ACM SIGCOMM Computer Communication Review · doi:10.1145/3138808.3138810 · Cited by 411 · see citing works
Hussain, F. A., Hassan, S. A., Hussain, R., & Hossain, E. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2964534 · Cited by 402 · see citing works
“allocation in wireless systems is presented in [78] where the authors used DRL algorithms to manage resources efficiently in the presence of interference and noise.”
MethodOptimization of global production scheduling with deep reinforcement learning
Waschneck, B., Reichstaller, A., Belzner, L., Altenmüller, T., Bauernhansl, T., Knapp, A., & Kyek, A. (2018) · Procedia CIRP · doi:10.1016/j.procir.2018.03.212 · Cited by 368 · see citing works
Deep Learning Meets SAR: Concepts, models, pitfalls, and perspectives
Zhu, X. X., Montazeri, S., Ali, M., Hua, Y., Wang, Y., Mou, L., Shi, Y., Xu, F., & Bamler, R. (2021) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2020.3046356 · Cited by 403 · see citing works
Device Placement Optimization with Reinforcement LearningPreprint
Mirhoseini, A., Pham, H., Le, Q. V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Bengio, S., & Dean, J. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1706.04972 · Cited by 504 · see citing works
“For example, Mao et al. (2016) pro-pose to train a resource management algorithm with policy gradients.”
BackgroundExplainability in deep reinforcement learning
Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106685 · Cited by 337 · see citing works
Deep Reinforcement Learning for Internet of Things: A Comprehensive Survey
Chen, W., Qiu, X., Cai, T., Dai, H., Zheng, Z., & Zhang, Y. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3073036 · Cited by 318 · see citing works
“With one step further, Mao et al. in [130] considered dispatching tasks with multiple resource demands and proposed a novel policy gradient method that promotes the monotonic improvement of policy iteration.”
BackgroundResource Allocation Based on Deep Reinforcement Learning in IoT Edge Computing
Xiong, X., Zheng, K., Lei, L., & Hou, L. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.2986615 · Cited by 301 · see citing works
“time [35].”
Multi-Objective Workflow Scheduling With Deep-Q-Network-Based Multi-Agent Reinforcement Learning
Wang, Y., Liu, H., Zheng, W., Xia, Y., Li, Y., Chen, P., Guo, K., & Xie, H. L. (2019) · IEEE Access · doi:10.1109/access.2019.2902846 · Cited by 288 · see citing works
Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement LearningPreprint
Zhang, C., Song, W. W., Cao, Z., Zhang, J., Tan, P. S., & Xu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12367 · Cited by 293 · see citing works
“In [23], a DRL agent is proposed to learn job scheduling policies for a compute cluster.”
BackgroundChen, L., Lingys, J., Chen, K., & Liu, F. (2018) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3230543.3230551 · Cited by 283 · see citing works
Deep reinforcement learning: a survey
Wang, H.-N., Liu, N., Zhang, Y., Feng, D., Huang, F., Li, D., & Zhang, Y. (2020) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1900533 · Cited by 286 · see citing works
“Tackling multi-resource cluster scheduling with a PG algorithm (Mao et al., 2016) optimizes various objectives like average job slowdown or completion time in an online manner with dynamic job arrivals, and validates the approach via simulation.”
MethodQi, Q., Wang, J., Ma, Z., Sun, H., Cao, Y., Zhang, L., & Liao, J. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2894437 · Cited by 268 · see citing works
“Mao et al. [22] find the optimal resource management policy by DRL.”
BackgroundTuli, S., Ilager, S., Ramamohanarao, K., & Buyya, R. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.3017079 · Cited by 262 · see citing works
A Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 264 · see citing works
“Reading maps [329], face hallucination [330], view planning [331] and semantic parsing [332] are other applications of RL in computer vision.”
MethodDirect shape optimization through deep reinforcement learning
Viquerat, J., Rabault, J., Kuhnle, A., Ghraieb, H., Larcher, A., & Hachem, E. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.110080 · Cited by 251 · see citing works
Deep Reinforcement Learning for Resource Allocation in V2V Communications
Ye, H., & Li, G. Y. (2018) · 2018 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2018.8422586 · Cited by 222 · see citing works
Li, C. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9781351006620-6 · Cited by 251 · see citing works
Migration Modeling and Learning Algorithms for Containers in Fog Computing
Tang, Z., Zhou, X., Zhang, F., Jia, W., & Zhao, W. (2018) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2018.2827070 · Cited by 230 · see citing works
“[58] use the deep reinforcement learning in network resource scheduling.”
MethodUsing GANs for Sharing Networked Time Series Data
Lin, Z., Jain, A., Wang, C., Fanti, G. C., & Sekar, V. (2020) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3419394.3423643 · Cited by 232 · see citing works
“Data-driven techniques [51] are central to networking and systems research (e.g., [11, 16, 23, 48, 60, 60, 71, 74, 83, 104]).”
BackgroundCellular Network Traffic Scheduling With Deep Reinforcement Learning
Chinchali, S., Hu, P., Chu, T., Sharma, M., Bansal, M., Misra, R., Pavone, M., & Katti, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11339 · Cited by 218 · see citing works
“A recent application of RL to job scheduling in internet data centers is deepRM (Mao et al. 2016).”
BackgroundMethodValadarsky, A., Schapira, M., Shahaf, D., & Tamar, A. (2017) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3152434.3152441 · Cited by 223 · see citing works
Routing optimization meets Machine Intelligence: A perspective for the future network
Dai, B., Cao, Y., Wu, Z., Dai, Z., Yao, R., & Xu, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.06.093 · Cited by 215 · see citing works
Reinforcement learning for iot security: A comprehensive survey
Uprety, A., & Rawat, D. B. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3040957 · Cited by 214 · see citing works
“Researchers have proposed many DRL approaches with its application ranging from control [15], resource management [16], [17], robotics [18], [19] and many more.”
BackgroundWang, Y., Wang, K., Huang, H., Miyazaki, T., & Guo, S. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2883991 · Cited by 213 · see citing works
“Many existing studies focused on only a particular type of network or type of task [21], [22].”
BackgroundA Graph Neural Network-Based Digital Twin for Network Slicing Management
Wang, H., Wu, Y., Min, G., & Miao, W. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3047843 · Cited by 207 · see citing works
“Researchers are using Deep Neural Networks (DNN) to model complex network behaviours [9] and develop decision-making strategies based on Deep Reinforcement Learning [10].”
MethodLearning to Perform Local Rewriting for Combinatorial OptimizationPreprint
Chen, X., & Tian, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00337 · Cited by 202 · see citing works
“Following DeepRM [33], we use the average job slowdown ηj ≡ (Cj −Aj)/Tj ≥ 1 as our evaluation metric.”
BackgroundMethodInfluentialDeep multiagent reinforcement learning: challenges and directions
Wong, A., Bäck, T., Kononova, A. V., & Plaat, A. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10299-x · Cited by 201 · see citing works
“2015), resource management (Wen et al. 2015; Mao et al. 2016), indoor robot navigation (Zhu et al.”
BackgroundCFR-RL: Traffic Engineering With Reinforcement Learning in SDN
Zhang, J., Ye, M., Guo, Z., Yen, C.-Y., & Chao, H. J. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3000371 · Cited by 200 · see citing works
“ach state s t, CFR-RL would select K critical flows. Given that there are total N „N 1”flows in a network with N nodes, this RL problem would require a large action space of size CK N„ 1” . Inspired by [27], we define the action space as {0, 1, ..., „N „N 1”” 1} and allow the agent to sample K different actions in each t…”
BackgroundA deep reinforcement learning approach for chemical production scheduling
Hubbs, C. D., Li, C., Sahinidis, N. V., Grossmann, I. E., & Wassick, J. M. (2020) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2020.106982 · Cited by 199 · see citing works
Jamil, B., Ijaz, H., Shojafar, M., Munir, K., & Buyya, R. (2022) · ACM Computing Surveys · doi:10.1145/3513002 · Cited by 198 · see citing works
“In 2016, Mao et al. [124] are the first ones who claim that a system itself can learn to manage resources.”
BackgroundMachine learning (ML)-centric resource management in cloud computing: A review and future directions
Khan, T., Tian, W., Zhou, G., Ilager, S., Gong, M., & Buyya, R. (2022) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2022.103405 · Cited by 205 · see citing works
Zhao, Y., Li, Y., Zhang, X., Geng, G., Zhang, W., & Sun, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2928564 · Cited by 176 · see citing works
“DeepRM [52] was presented as a solution that translates the problem of packing tasks with multiple resource demands into a learning problem.”
BackgroundA Learning-based Iterative Method for Solving Vehicle Routing Problems
Hao, L., Zhang, X., & Yang, S. (2020) · International Conference on Learning Representations · Cited by 175 · see citing works
Lee, S.-S., & Lee, S. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2996213 · Cited by 171 · see citing works
“Mao et al. [28] employed the REINFORCE algorithm [36] for a resource allocation problem with a discrete action space.”
MethodGawłowicz, P., & Zubow, A. (2019) · ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM) · doi:10.1145/3345768.3355908 · Cited by 167 · see citing works
“…a boom in the usage of machine learning in general and reinforcement learning (RL) in particular for the optimization of communication and networking systems ranging from scheduling [2, 6], resource management [18], congestion control [14, 15, 29], routing [1] and adaptive video streaming [19].”
BackgroundDeep reinforcement learning based control for Autonomous Vehicles in CARLA
Pérez-Gil, Ó., Barea, R., López-Guillén, E., Bergasa, L. M., Gómez-Huélamo, C., Gutiérrez, R., & Díaz-Díaz, A. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11437-3 · Cited by 165 · see citing works
“…through AV architecture, dealing with different processes such as detection, Multi-Object Tracking (MOT) and environment prediction, or evaluating the current situation of the ego-vehicle to conduct the safest decision, for example making use of DRL algorithms for behavioural driving [31].”
MethodDeep and reinforcement learning for automated task scheduling in large‐scale cloud computing systems
Rjoub, G., Bentahar, J., Wahab, O. A., & Bataineh, A. S. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5919 · Cited by 156 · see citing works
Chen, Z., Hu, J. W., Min, G., Luo, C., & El-Ghazawi, T. A. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3132422 · Cited by 154 · see citing works
“Different from these work using value-based DRL algorithms, Mao et al. [20] leveraged a policy-based DRL algorithm (i.e., policy gradient (PG)) to handle the resource allocation problem in cloud datacenters.”
MethodDeep Learning-based Job Placement in Distributed Machine Learning Clusters
Bao, Y., Peng, Y., & Wu, C. (2019) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2019.8737460 · Cited by 148 · see citing works
“In this way, we use multiple inferences to effectively reduce action space [23] [22].”
MethodLiang, E., Zhu, H., Jin, X., & Stoica, I. G. (2019) · ACM Special Interest Group on Data Communication · doi:10.1145/3341302.3342221 · Cited by 146 · see citing works
“There are also many solutions that apply deep reinforcement learning to congestion control [6, 16, 64] and resource management [34].”
BackgroundInfluentialInternet of Things Connected Wireless Sensor Networks for Smart Cities
Ghazal, T. M., Hasan, M. K., Alzoubi, H. M., Alshurideh, M. T., Ahmad, M., & Akbar, S. S. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-12382-5_107 · Cited by 147 · see citing works
Liang, H., Wen, X., Liu, Y., Zhang, H., Li, Z., & Wang, L. (2020) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2020.101991 · Cited by 144 · see citing works
Reinforcement learning-assisted evolutionary algorithm: A survey and research opportunities
Song, Y., Wu, Y., Guo, Y., Yan, R., Suganthan, P. N., Zhang, Y., Witold Pedrycz, C., Das, S., Mallipeddi, R., Ajani, O. S., et al. (2024) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2024.101517 · Cited by 150 · see citing works
Peekaboo: Learning-Based Multipath Scheduling for Dynamic Heterogeneous Environments
Wu, H., Alay, Ö., Brunström, A., Ferlin, S., & Caso, G. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3000365 · Cited by 143 · see citing works
“Previous work has also considered using Deep RL (DRL) methods for decision-making problems, such as DeepRM [9] and Pensieve [10].”
Hu, X., Liu, S., Chen, R., Wang, W., & Wang, C. (2018) · IEEE Communications Letters · doi:10.1109/lcomm.2018.2844243 · Cited by 142 · see citing works
“DRL as an effective online method have achieved many success in game playing [5] and resource management [6].”
MethodDeep reinforcement learning for semiconductor production scheduling
Waschneck, B., Reichstaller, A., Belzner, L., Altenmüller, T., Bauernhansl, T., Knapp, A., & Kyek, A. (2018) · ASMC proceedings · doi:10.1109/asmc.2018.8373191 · Cited by 142 · see citing works
Jayanetti, A., Halgamuge, S. K., & Buyya, R. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.06.012 · Cited by 143 · see citing works
“Therefore, in each actual time-step, we use the DRL agent multiple times to determine the nodes in which all tasks in RTQ are to be scheduled [11].”
MethodEfficient Data Structures and Algorithms for Cloud Computing Platforms
Aladiyan, A. (2024) · International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) · doi:10.1109/icacite60783.2024.10617203 · Cited by 135 · see citing works
“[9] Bloom Filters Space-efficient data structures that tell you whether an element might be in a set.”
BackgroundQian, B., Su, J., Wen, Z., Jha, D. N., Li, Y., Guan, Y., Puthal, D., James, P., Yang, R., Zomaya, A. Y., et al. (2021) · ACM Computing Surveys · doi:10.1145/3398020 · Cited by 132 · see citing works
“Another task formulated in discrete action space can be found in Reference [182], where the action space is the “schedule of the job at i-th slot.”
BackgroundInfluentialZhan, Y., Li, P., & Guo, S. (2020) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps47924.2020.00033 · Cited by 134 · see citing works
“Towards this direction, DeepRM [46] leveraged DRL to solve the online multi-resource job placement problem and can achieve better performance as compared with the heuristic algorithms.”
MethodLeveraging Deep Reinforcement Learning for Traffic Engineering: A Survey
Xiao, Y., Liu, J., Wu, J., & Ansari, N. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3102580 · Cited by 137 · see citing works
Gazori, P., Rahbari, D., & Nickray, M. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.09.060 · Cited by 130 · see citing works
Liu, S., Hu, X., & Wang, W. (2018) · IEEE Access · doi:10.1109/access.2018.2809581 · Cited by 129 · see citing works
“The emerging DRL method shows great prospect in sequential decision making problems [12]–[15] and has been applied in many dynamic resource management problems [16]–[20].”
MethodDeep Reinforcement Learning Based Optimization Algorithm for Permutation Flow-Shop Scheduling
Pan, Z., Wang, L., Wang, J., & Lu, J. (2021) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3098354 · Cited by 127 · see citing works
“…improvement heuristics for TSP and VRP. Xin et al. [39] presented a novel DRL method by using multi-decoder attention model with embedding glimpse to solve VRP. Mao et al. [40] built management systems with a designed DeepRM based on deep RL (DRL) to learn resource managementdirectlyfromexperience.”
MethodTran‐Dang, H., Bhardwaj, S., Rahim, T., Musaddiq, A., & Kim, D. (2022) · Journal of Communications and Networks · doi:10.23919/jcn.2021.000041 · Cited by 134 · see citing works
Ng, J. S., Lim, W. Y. B., Luong, N. C., Xiong, Z., Asheralieva, A., Niyato, D. T., Leung, C., & Miao, C. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3091684 · Cited by 143 · see citing works
“Moreover, for the future works, the dynamics of the system may be captured using Deep Reinforcement Learning based resource allocation [229].”
Islam, M. T., Karunasekera, S. A., & Buyya, R. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3124670 · Cited by 125 · see citing works
“There are lots of existing works [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19] that focused on various SLA objectives.”
BackgroundMethodHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · see citing works
“autonomous vehicles [27,28] smart grids [29], fleet management [30], resource management [31,32] and recommender systems [33].”
BackgroundAdaptable and Data-Driven Softwarized Networks: Review, Opportunities, and Challenges
Kellerer, W., Kalmbach, P., Blenk, A., Basta, A., Martin Reisslein, B., & Schmid, S. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2895553 · Cited by 123 · see citing works
“[102] H.”
Cloud Resource Scheduling With Deep Reinforcement Learning and Imitation Learning
Guo, W., Tian, W., Ye, Y., Xu, L., & Wu, K. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3025015 · Cited by 122 · see citing works
“Nevertheless, depending on different machine learning techniques used in the solution, research in this domain can be classified in different ways, including: 1) bioinspired solutions [28]; 2) Q -learning-based solutions [29]–[31]; and 3) deep RL-based solutions [17], [32].”
BackgroundMethodInfluentialDeep Reinforcement Scheduling for Mobile Crowdsensing in Fog Computing
Li, H., Ota, K., & Dong, M. (2019) · ACM Transactions on Internet Technology · doi:10.1145/3234463 · Cited by 118 · see citing works
“—DeepRM: a resource management method with deep reinforcement learning [31].”
BackgroundMethodSellami, B., Hakiri, A., Yahia, S. B., & Berthou, P. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.108957 · Cited by 120 · see citing works
“[30] introduced a DeepRM framework to build autonomous and intelligent systems that learn to manage resources from their own experience directly.”
BackgroundLiu, W., Cai, J., Chen, Q. C., & Wang, Y. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2020.102865 · Cited by 120 · see citing works
AI-Enhanced Cloud-Edge-Terminal Collaborative Network: Survey, Applications, and Future Directions
Gu, H., Zhao, L., Han, Z., Zheng, G., & Song, S. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3338153 · Cited by 116 · see citing works
AI-Based Resource Provisioning of IoE Services in 6G: A Deep Reinforcement Learning Approach
Sami, H., Otrok, H., Bentahar, J., & Mourad, A. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3066625 · Cited by 114 · see citing works
“Aware of the above limitations and cooping with 1) the Deep Reinforcement Learning (DRL) advancements in the resource management field [21], and 2) the potential of having an AI fostered environment in 6G [22], exploring the use of DRL as a resource management solution is promising.”
BackgroundDDPG-Based Joint Time and Energy Management in Ambient Backscatter-Assisted Hybrid Underlay CRNs
Zheng, K., Jia, X., Chi, K., & Liu, X. (2022) · IEEE Transactions on Communications · doi:10.1109/tcomm.2022.3221422 · Cited by 113 · see citing works
“Secondly, utilizing DRL can address complex tasks, and keeps graceful performance under varying conditions by continuing learning [26], [27].”
MethodThermal Prediction for Efficient Energy Management of Clouds Using Machine Learning
Ilager, S., Ramamohanarao, K., & Buyya, R. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3040800 · Cited by 111 · see citing works
Toward Packet Routing With Fully Distributed Multiagent Deep Reinforcement Learning
You, X., Li, X., Xu, Y., Feng, H., Zhao, J., & Yan, H. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.3012832 · Cited by 109 · see citing works
“Recent successful applications include cloud resource allocation [21], adaptive bitrate video streaming [22], and cellular scheduling [23].”
BackgroundMachine-Learning-Assisted Security and Privacy Provisioning for Edge Computing: A Survey
Singh, S., Sulthana, A. R., Shewale, T., Chamola, V., Benslimane, A., & Sikdar, B. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3098051 · Cited by 108 · see citing works
“Reinforcement algorithms are capable of handling certain problems that require heuristic human intelligence and can take the place of human intelligence [39].”
BackgroundDL2: A Deep Learning-Driven Scheduler for Deep Learning Clusters
Peng, Y., Bao, Y., Chen, Y., Wu, C., Meng, C., & Lin, W. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3052895 · Cited by 112 · see citing works
“, experience replay, actor-critic) as [14], [16], and also uses the training data generated by an analytical performancemodel [12] in a simulation environment.”
BackgroundMethodInfluentialHossain, M. A., Noor, R. M., Yau, K. A., Azzuhri, S. R., Z’aba, M. R., & Ahmedy, I. (2020) · IEEE Access · doi:10.1109/access.2020.2989870 · Cited by 105 · see citing works
“Concerning the research question, ‘‘Can systems learn to manage resources on their own?’’, the answer can be found in [292], in which an experiment was conducted using deep RL and DNN.”
BackgroundLee, J., Lee, Y. C., & Kim, J. T. (2020) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2020.10.013 · Cited by 106 · see citing works
RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement Learning
Zhang, D., Dai, D., He, Y., Bao, F. S., & Xie, B. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41405.2020.00035 · Cited by 107 · see citing works
“Examples include resource scheduling and task provisioning, such as DeepRM [34, 39], Decima [28], and RRL [40]; resource configuration tuning [41–43]; file system tuning [44] and performance prediction [45, 46].”
BackgroundKabir, H., Tham, M., & Chang, Y. C. (2023) · Digital Communications and Networks · doi:10.1016/j.dcan.2023.05.006 · Cited by 119 · see citing works
Yao, H., Mai, T., Xu, X., Zhang, P., Li, M., & Liu, Y. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2859480 · Cited by 101 · see citing works
“There is a large amount of literature on applying supervised learning to solve specific problems for a variety of network applications, such as resource allocations [13], Web services [14], [15], Internet telephony [16], and video streaming [17].”
BackgroundMulti-agent reinforcement learning for resources allocation optimization: a survey
Hady, M. A., Hu, S., Pratama, M., Cao, Z., & Kowalczyk, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11340-5 · Cited by 120 · see citing works
“In RAO, this allows the agent to dynamically allocate resources based on the current system state and its expected impact on future performance Mao et al. (2016).”
MLCAD: A Survey of Research in Machine Learning for CAD Keynote Paper
Rapp, M., Amrouch, H., Lin, Y., Yu, B., Pan, D. Z., Wolf, M., & Henkel, J. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3124762 · Cited by 99 · see citing works
“Mao et al. [121] used policy-based RL to decide when to schedule jobs with the goal of maximizing the performance.”
BackgroundLearning and Generating Distributed Routing Protocols Using Graph-Based Deep Learning
Geyer, F., & Carle, G. (2018) · doi:10.1145/3229607.3229610 · Cited by 96 · see citing works
“In this context, machine learning has often been proposed and applied to various tasks such as routing [3, 25, 28], computing resource management [16] or packet scheduling [4].”
MethodReinforcement Learning for Slicing in a 5G Flexible RAN
Raza, M. R., Natalino, C., Öhlén, P., Wosinska, L., & Monti, P. (2019) · Journal of Lightwave Technology · doi:10.1109/jlt.2019.2924345 · Cited by 97 · see citing works
“The RL agent is modeled as a stochastic policy network (PN) [Mao16], which, in turn, uses an artificial neural network (ANN) to represent its stochastic policy.”
Intelligent VNF Orchestration and Flow Scheduling via Model-Assisted Deep Reinforcement Learning
Gu, L., Zeng, D., Li, W., Guo, S., Zomaya, A. Y., & Jin, H. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2959182 · Cited by 97 · see citing works
“…learning, including DRL, has been successfully applied as an alternative promising means to address many issues in computer networks, e.g., TCP congestion control [8], workload balancing [9], SDN routing [10], resource allocation [11], wireless signal calibration [12], and traffic optimization [13].”
MethodDynamic Jobshop Scheduling Algorithm Based on Deep Q Network
Zhao, Y., Wang, Y., Tan, Y., Zhang, J., & Yu, H. (2021) · IEEE Access · doi:10.1109/access.2021.3110242 · Cited by 97 · see citing works
“Mao et al. [29] proposed a multi-resource cluster scheduler called DeepRM for scheduling problems, and built a system that can learn scheduling strategies directly from previous scheduling experience.”
BackgroundYang, Z., Nguyen, P., Jin, H., & Nahrstedt, K. (2019) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00021 · Cited by 94 · see citing works
“problem of packing tasks with multiple resource demands into a deep reinforcement learning problem [22].”
BackgroundEnergy-Efficient Artificial Intelligence of Things With Intelligent Edge
Zhu, S., Ota, K., & Dong, M. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3143722 · Cited by 97 · see citing works
- Authors
- Mao, H., Alizadeh, M., Menache, I., & Kandula, S.
- Affiliations
- Massachusetts Institute of Technology (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 15th ACM Workshop on Hot Topics in Networks
- Publisher
- ACM
- Published
- 2016-11-09
- Pages
- 50-56
- OpenAlex
- W2546571074
- Cited by
- 1,353
- References
- 41
- Citations by year
- 2026: 171 · 2025: 181 · 2024: 98 · 2023: 140 · 2022: 151 · 2021: 191 · 2020: 180 · 2019: 155 · 2018: 62 · 2017: 24
- FWCI
- 68.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Workshop on Hot Topics in Networks (HotNets) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Keywords
- resource management, deep reinforcement learning, task scheduling, resource allocation, workload modeling, online decision making, experiential learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
