Resource Management with Deep Reinforcement Learning

“[30] proposed DeepRM, a solution based on deep reinforcement learning.”

“, for load balancing, packet classification, resource scheduling, and intrusion detection) in datacenter networks [19, 48, 59, 89, 100].”

“These algorithms have been used for various scheduling and resource allocation tasks in data centers [53], wireless networks [54], edge computing [55] or vehicular networks [56].”

“The DNN was trained on a sequence of jobs to minimize average job slowdown (job’s duration in the system divided by its actual processing time) using the open source code [10].”

“Using RL, researchers can anticipate the attributes of upcoming tasks which in turn can lead to various near optimum solutions [18], [19], [20].”

“RL exceeds human performance in domains ranging from resource management [31], traf fi c light control [32], playing sophisticated games such as chess [33], and Atari [34], to chemistry [35].”

“potential of reinforcement learning (RL) on network resource management problems in general [6], and V2V resource management in particular [7], we implement a centralized scheduling algorithm that learns by using the information available in V2X environment, such as the occupancy of radio resources.”

“Machine learning techniques have been successfully applied in many of the control or optimization problems in the networking field, including cluster scheduling [14, 5], adaptive video streaming [13], congestion control [9], and traffic engineering [24, 3].”

“In addition, with reference to [13], at each time step, the time stops until the void action or invalid action is selected by the scheduler.”

“Besides, when the decision process is stochastic, action-value methods have no natural way of finding stochastic optimal policies, whereas policy approximating methods can [15], [16].”

“Training Algorithm for DRL-R’s Policy Network Since we represent the policy as a CNN (called policy network) which takes as input the collection of images described above, and outputs a probability distribution over all possible actions, we train the CNN using a variant of the REINFORCE algorithm described as shown in…”

“(12), for each sequence in each training iteration, the Δ θ can be calculated as [24] where b il = 1 /I i v il is a baseline.”

“RL approaches are especially well suited to RM systems because these systems produce repetitive results, providing ample opportunities for learning from training data.(6) Before introducing the taxonomy of RL algorithms, it is essential to know a few terms.”

“Reference [39] also concludes the paper with a suggestion to use the machine learning in admission control in real time and in resoiuce allocation like in [40].”

“Mao et al. [18] pioneered the application of RL to resource management in cloud computing, demonstrating that deep reinforcement learning could outperform heuristic-based schedulers.”

“Still others have considered dynamic and global nature of the resource allocation problem [6, 28, 8, 27, 5, 13, 14].”

“, cloud computing and server cluster operations [5]–[7].”

“Performance SLOs.”

“Inspired by this and by some prior work done in DRL for CPU/memory cluster resource allocation [7], we capture the channel and traffic states in the system via images as shown in Figure 2.”

“Inspired by MARL and the work in [28], we develop a novel MARL architecture to solve the optimal iterative contract problem.”

“DRL techniques dynamically interact with the environment to facilitate decision-making [10], [11], [12].”

“Another domain of application of ML, like exemplified in [25], is resource allocation.”

“Particularly, a deep neural network can be used as a function approximator to represent the optimal policy [24].”

“In [18–22], the author used the method of deep reinforcement learning to optimize the resource allocation under different application scenarios.”

“There exist several works on applying DRL for scheduling tasks in the various domains such as cloud computing, networks, and manufacturing systems [8]–[11], [15]–[18].”

“DeepRM [106] Device Placement [118] Decima [108]”

“Although the actual reward of the current scheduling decision can be known only when the scheduled tasks are completed, as in [26], [39], we wait for a time period after the scheduling decision is made at time t 0 .”

“Mao et al. [30] managed the diverse computing resources using a deep reinforcement learning method.”

“In recent years, self-learning approaches, particularly reinforcement learning, have become popular in various areas of networking such as traffic engineering and load balancing [26], [27], [28], routing [29], [30], service coordination [31], [32], [33], and resource management [34], [35], [36].”

“[8] utilized deep reinforcement learning (DRL) algorithm for adaptively solving resource allocation problems.”

“Besides, there is a new trend to use deep reinforcement learning (DRL) techniques to learn dynamic scheduling policies [8, 21, 22].”

“Other examples where RL has aided the solution of real-world problems include resource management [23], robotics [24] and chemistry [25], amongst myriad others.”

“Moreover, for the future works, the dynamics of the system may be captured using Deep Reinforcement Learning based resource allocation [220].”

“In addition to traffic prediction, many other tasks in performance management have seen several proposed ML based solutions: Traffic management in cloud and mobile edge com‐ puting [45, 95], network resource management and allocation [8, 46, 47, 96], QoS assurance [97, 98], and congestion control [48, 49].”

“So far, there have been no standardized methods for designing neural networks, but existing research provided some reference [5,33].”

“Classical resource allocation algorithms, such as the first-in-first-out method and shortest-job-first method, assume that resource demands and length of jobs are known in advance [9] .”

“Applications of deep reinforcement learning or deep learning in general to resource management exist (cf. [9] or [10] for radar).”

“Since the objective is that of maximizing the expected cumulative discounted reward, the gradient of the objective function is given as follows [154]: (28) where Q π θ ( s , a ) is the expected cumulative discounted reward obtained from deterministically selecting an action a under the guidance of…”

“Schedulers based on deep reinforcement learning [49, 50] may have thousands or millions of such parameters and they must be optimized on large amounts of synthetic data.”

“However, due to higher uncertainty in quantifying the advantage of proactive transmission, we adopt negative rewards to discourage situations with transmission progress falling behind [50].”

“Deep reinforcement learning has also been applied in resource management problems for systems and networking [60]: the decision making tasks where appropriate solutions are taken depend on understanding the workload and environment experience.”

“[15] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula.”

“Compared to the state-of-the-art reinforcement learning methods, RRL can reduce the average latency up to 79.0% on CPU-based infrastructure and up to 69.3% on GPU-based infrastructure compared to state-of-the-art approaches DeepRM[38] and CAPES[37].”

“We flatten the collected state into a fully connected neural network architecture (this is an approach similar to [12] and [30]).”

“Both [14] and [8] rely on synthetic data to train their approach.”

“, cores) fs and fl follow an Uniform Distribution, k ∈ [160, 200], fs ∈ [25, 32], fl ∈ [6, 8], respectively.”

“All the solutions presented in [34], [35], [36], [37] share the common assumption of having access to detailed taskrelated information (i.”

“After the success of the DeepRM [27] there is a line of research pursuing the problem of resource allocation with RL to optimize classic system problems.”

“DRL has achieved success in many networking and resource management tasks [39, 40].”

“For example, Mao et al. [26] applied RL in resource management and scheduling, demonstrating its ability to enhance scheduling performance.”

“In existing HPC job scheduling research, the scheduling goals can be categorized into three main areas: maximizing resource utilization [4, 24], optimizing time-related objectives [4, 6, 19–21, 23, 37, 38], and optimizing mixed objectives of utilization and time [5, 8, 9, 12, 16].”

“With the development of RL technology, Pensieve [7], proposed in 2017, as a combination of traditional streaming media transmission and RL, can utilize more feature information, thus choosing bitrate more flexibly [5, 6, 8].”

“Similar to the prior work [27], we assume that the information related to each job is known upon arrival; more specifically, the information mainly includes the number of child tasks and the details of each task (e.g., the length and storage location of pending data, number of CPU instructions).”

“We validate our proposed Deep Reinforcement Learning approach by implementing our model on top of DeepRM, which was developed by Mao [5] for resource scheduling using deep learning, and achieve an average training accuracy of 82.”

“Thus, DDPG for building DRL-Flow was employed; DDPG's detailed algorithm is presented in.(19) Because the policy network is represented as a CNN that accepts as input a collection of states s and outputs a probability distribution over all possible actions, this paper used realistic DCN mix-flow traffic (in our case,…”

“…intelligent scheduling framework for a direct comparison, we opt to implement the state-of-the-art reinforcement learning approaches for tuning parallelism configuration in our scheduling framework: CAPES [15] and DeepRM [14], as they are the closest approaches for the online ML-serving scheduling.”

“DeepRM [11] should be the first work to apply DRL on job scheduling.”

“The deep neural networks in RL, commonly referred to as deep Q-learning [56], were manually constructed in previous studies [48,51,57].”

“For instance, this optimizer could be implemented with a Reinforcement Learning algorithm that learns from previous verifier requests [36] in order to reduce the computation delay.”

“DeepRM [9] is presented for the resource allocation in the data center, which translates the problem of packing tasks with multiple resource demands into a deep reinforcement learning problem.”

“[15] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula.”

“Reinforcement learning could be applied to efficiently manage computer clusters [33], traffic lights [34], robotics [35], and chemistry [36].”

“…in sequential decision-making problems (Mnih et al., 2013, 2015, 2016; Li L et al., 2016; Silver et al., 2017; Wang et al., 2020), and a variety of RL-based solutions have been applied in dynamic control and other domains (Mao et al., 2016; Liu et al., 2017; Xu et al., 2017; He Y et al., 2018).”

“Recently ML [32] and AI models [33] have been successfully applied to task scheduling through real-time intelligent decision making [27].”

“DRL in other areas: Besides load balancing, RL or DRL has been applied to various areas including congestion control [26], flow scheduling [14], resource allocation [33, 35], video streaming [34], etc.”

“In recent years, DRL has received significant attention in the areas of workload balancing [30], network resource management [31], routing [32], and other problems.”

“The REINFORCE algorithm [33,34] iteratively updates θ using the stored ( s , a , r ) data set, adjusting the policy network to output a probability distribution of actions that favors the selection of actions maximizing the long-term return H .”

“For example, deep reinforcement learning has been used to study cloud resource management [19–21].”

“[14] employs deep learning strategy in reinforcement learning to accomplish resource management in a cluster from gradually accumulated experience and the result called DeepRM is comparable to state-of-the-art heuristics.”

“5 RL with policy represented via NN [27] through the random inversion of a strategy’s codes.”

“A DRL dependent approach is invented in [9] to locate the issue of job scheduling with demands of multiple resources, these type of problem raised in computing system while managing resources dynamically is hectic, where the motivation is to diminish the reward function and the average job stagnation is depending on t…”

“, which commonly calculates the gradient as [25], [34] D E u X g t r t h i ¼ E p u ½ D u log p u ð s; a Þ Q p u ð s; a Þ(cid”

“…the well-known policy gradient algorithms [24, 31, 32] and deep Q-networks [25], have shown superior performance and great potential in many difficult real-world problems, such as the Go game [33, 34], locomotive continuous control problems [21], resource management [23], and robotics [20].”

“In the work of Mao et al, 14 authors applied the Deep RL technique in machine intelligence, where the proposed system learns directly the resource management between various electronic devices by accumulating the experiences received within a cluster.”

“Associating the resources with the channels in our work, we note that the experiment presented in [48] uses only two resources and takes 80 s per iteration to converge in a multi-threaded implementation on a 24-core CPU server.”

“Recent studies report that many production analytics workloads are recurring, such as running the same queries periodically on new data [17], [19], [33], [52], [67].”

“Reinforcement learning has also been tested for resource allocation decision-makings in project management demonstrating its performance (Gai and Qiu, 2018; Huang et al., 2011; Mao et al., 2016).”