A Double Deep Q-Learning Model for Energy-Efficient Edge Scheduling
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Reducing energy consumption is a vital and challenging problem for the edge computing devices since they are always energy-limited. To tackle this problem, a deep Q-learning model with multiple DVFS (dynamic voltage and frequency scaling) algorithms was proposed for energy-efficient scheduling (DQL-EES). However, DQL-EES is highly unstable when using a single stacked auto-encoder to approximate the Q-function. Additionally, it cannot distinguish the continuous system states well since it depends on a Q-table to generate the target values for training parameters. In this paper, a double deep Q-learning model is proposed for energy-efficient edge scheduling (DDQ-EES). Specially, the proposed double deep Q-learning model includes a generated network for producing the Q-value for each DVFS algorithm and a target network for producing the target Q-values to train the parameters. Furthermore, the rectified linear units (ReLU) function is used as the activation function in the double deep Q-learning model, instead of the Sigmoid function in QDL-EES, to avoid gradient vanishing. Finally, a learning algorithm based on experience replay is developed to train the parameters of the proposed model. The proposed model is compared with DQL-EES on EdgeCloudSim in terms of energy saving and training time. Results indicate that our proposed model can save average 2%-2.4% energy and achieve a higher training efficiency than QQL-EES, proving its potential for energy-efficient edge scheduling.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Compared to DQL, the experiment results indicate that Double-DQL can save more energy and achieve higher training efficiency.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [257]
Citations
Cites
Tensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 1×
“Furthermore, we will investigate the use of tensor decompositions and tensor-train networks [30], [31], [32], [33] to improve the training efficiency of the presented double deep Q-learning model in the future work.”
MethodEdge computing: Vision and challenges
Shi, W., Cao, J., Zhang, Q., Li, Y., & Xu, L. (2016) · IEEE Internet of Things Journal · doi:10.1109/jiot.2016.2579198 · Cited by 8,733 · cited 3×
“Second, cloud computing cannot ensure the privacy protection sometimes in Internet of Things and cyber-physical-social systems [14].”
BackgroundResultSelf-citation(2011) · Cited by 6,993
MiBench: A free, commercially representative embedded benchmark suite
Guthaus, M. R., Ringenberg, J. S., Ernst, D. J., Austin, T. M., Mudge, T., & Brown, R. B. (2001) · doi:10.1109/wwc.2001.15 · Cited by 3,103
Oseledets, I. V. (2011) · SIAM Journal on Scientific Computing · doi:10.1137/090752286 · Cited by 2,636
MiBench: A free, commercially representative embedded benchmark suite
Guthaus, M. R., Ringenberg, J. S., Ernst, D., Austin, T. M., Mudge, T., & Brown, R. B. (2005) · Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538) · doi:10.1109/wwc.2001.990739 · Cited by 1,588 · cited 1×
“We use one MiBench as the experimental taskset which includes a set of commercially representative embedded programs [29].”
MethodA survey on deep learning for big data
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.006 · Cited by 1,226
Self-citationReal-time dynamic voltage scaling for low-power embedded operating systems
Pillai, P., & Shin, K. G. (2001) · Proceedings of the eighteenth ACM symposium on Operating systems principles · doi:10.1145/502034.502044 · Cited by 1,138 · cited 2×
“Another two typical DVFS algorithms are CC and LA [19].”
BackgroundMethodPower-aware scheduling for periodic real-time tasks
Aydin, H., Melhem, R., Mosse, D., & Mejia-Alvarez, P. (2004) · IEEE Transactions on Computers · doi:10.1109/tc.2004.1275298 · Cited by 484
Zhou, Z., Dong, M., Ota, K., Wang, G., & Yang, L. T. (2016) · IEEE Internet of Things Journal · doi:10.1109/jiot.2015.2497712 · Cited by 272 · cited 1×
“Internet of Things integrate many physical and sensing devices including RFID (Radio Frequency Identification), camera and so on to the cyber space while cyber-physical-social systems combine the human behaviors and social relations to the Internet of Things [4], [5].”
BackgroundSelf-citationAn Efficient Deep Learning Model to Predict Cloud Workload for Industry Informatics
Zhang, Q., Yang, L. T., Yan, Z., Chen, Z., & Li, P. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2808910 · Cited by 231
Self-citationReal-Time Locating Systems Using Active RFID for Internet of Things
Zhang, D., Yang, L. T., Chen, M., Zhao, S., Guo, M., & Zhang, Y. (2016) · IEEE Systems Journal · doi:10.1109/jsyst.2014.2346625 · Cited by 217 · cited 1×
“RECENT years have seen the continuous growth of Internet of Things and cyber-physical-social systems as a result of the great development of network techniques, especially wireless sensor networks [1], [2], [3].”
BackgroundSelf-citationEdgeCloudSim: An environment for performance evaluation of Edge Computing systems
Sonmez, C., Ozgovde, A., & Ersoy, C. (2017) · 2017 Second International Conference on Fog and Mobile Edge Computing (FMEC) · doi:10.1109/fmec.2017.7946405 · Cited by 211
Deep Convolutional Computation Model for Feature Learning on Big Data in Internet of Things
Li, P., Chen, Z., Yang, L. T., Zhang, Q., & Deen, M. J. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2739340 · Cited by 199 · cited 1×
“lected data is transmitted to data centers for processing and analytics [6], [7], [8].”
Self-citationEfficient and Privacy-Preserving Outsourced Calculation of Rational Numbers
Liu, X., Choo, K.-K. R., Deng, R. H., Lu, R., & Weng, J. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2016.2536601 · Cited by 188 · cited 1×
“Most of typical data centers have been employing cloud computing to process and analyze big data since cloud computing can provide the powerful computing resources and the huge storage space [11], [12], [13].”
BackgroundEnergy-Efficient Scheduling for Real-Time Systems Based on Deep Q-Learning Model
Zhang, Q., Lin, M., Yang, L. T., Chen, Z., & Li, P. (2019) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2017.2743704 · Cited by 166 · cited 15×
“Specially, the limited energy supplying by the battery restricts the usage time and the service quality of edge computing devices for big data processing [15].”
BackgroundMethodSelf-citationUnderstanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
Shang, W., Sohn, K., Almeida, D., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05201 · Cited by 162 · cited 1×
“achieved great success in computer vision and natural language processing [26], [27].”
BackgroundZhang, Q., Yang, L. T., Chen, Z., & Li, P. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2701816 · Cited by 126
Self-citationZhang, Q., Yang, L. T., Chen, Z., Li, P., & Deen, M. J. (2017) · IEEE Internet of Things Journal · doi:10.1109/jiot.2017.2732735 · Cited by 122
Self-citationReal-Time Dynamic Voltage Loop Scheduling for Multi-Core Embedded Systems
Shao, Z., Wang, M., Chen, Y., Xue, C. J., Qiu, M., Yang, L. T., & Sha, E. H. ‐. (2007) · IEEE Transactions on Circuits and Systems II Analog and Digital Signal Processing · doi:10.1109/tcsii.2007.892215 · Cited by 111 · cited 1×
“Internet of Things and cyber-physical-social systems have been widely used in applications in industrial manufacturing and intelligent transportation with the big data analytics [9], [10].”
MethodSelf-citationPredicting Temporal Social Contact Patterns for Data Forwarding in Opportunistic Mobile Networks
Zhou, H., Leung, V. C. M., Zhu, C., Xu, S., & Fan, J. (2017) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2017.2740218 · Cited by 108
Multicloud-Based Evacuation Services for Emergency Management
Dong, M., Li, H., Ota, K., Yang, L. T., & Zhu, H. (2014) · IEEE Cloud Computing · doi:10.1109/mcc.2014.85 · Cited by 94 · cited 1×
“Most of typical data centers have been employing cloud computing to process and analyze big data since cloud computing can provide the powerful computing resources and the huge storage space [11], [12], [13].”
BackgroundSelf-citationHe, D., Chen, C., Chan, S., Bu, J., & Yang, L. T. (2013) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2012.2218562 · Cited by 78
Self-citationHybrid DVFS Scheduling for Real-Time Systems Based on Reinforcement Learning
Islam, F. M. M. U., & Lin, M. (2015) · IEEE Systems Journal · doi:10.1109/jsyst.2015.2446205 · Cited by 74 · cited 2×
“[21] presented an improved hybrid DVFS scheme based the double Q-learning approach.”
MethodResultSelf-citationAn Improved Deep Computation Model Based on Canonical Polyadic Decomposition
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2017.2701797 · Cited by 71
Self-citationConfident Information Coverage Hole Healing in Hybrid Industrial Wireless Sensor Networks
Deng, X., Tang, Z., Yang, L. T., Lin, M., & Wang, B. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2764038 · Cited by 64 · cited 1×
“Internet of Things and cyber-physical-social systems have been widely used in applications in industrial manufacturing and intelligent transportation with the big data analytics [9], [10].”
MethodSelf-citationA Tensor-Train Deep Computation Model for Industry Informatics Big Data Feature Learning
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2791423 · Cited by 59 · cited 1×
“Furthermore, we will investigate the use of tensor decompositions and tensor-train networks [30], [31], [32], [33] to improve the training efficiency of the presented double deep Q-learning model in the future work.”
MethodSelf-citationA privacy-preserving high-order neuro-fuzzy c-means algorithm with cloud computing
Li, P., Chen, Z., Yang, L. T., Zhao, L., & Zhang, Q. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.08.135 · Cited by 58
Self-citationHybrid power management in real time embedded systems: an interplay of DVFS and DPM techniques
Bhatti, M. K., Belleudy, C., & Auguin, M. (2011) · Real-Time Systems · doi:10.1007/s11241-011-9116-y · Cited by 43 · cited 1×
“In addition, some algorithms by combining DVFS and DPM techniques based on online learningwere also investigated in recent years [22].”
MethodQuality-Guaranteed Event-Sensitive Data Collection and Monitoring in Vibration Sensor Networks
Bhuiyan, M. Z. A., Wu, J., Wang, G., Chen, Z., Chen, J., & Wang, T. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2665463 · Cited by 42
Self-citationLin, X., Wang, Y., Chang, N., & Pedram, M. (2016) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2016.2523450 · Cited by 31
Efficient attribute based sequential aggregate signature for wireless sensor networks
Liu, X., Zhu, H., Ma, J., Li, Q., & Xiong, J. (2014) · International Journal of Sensor Networks · doi:10.1504/ijsnet.2014.066808 · Cited by 12 · cited 1×
“RECENT years have seen the continuous growth of Internet of Things and cyber-physical-social systems as a result of the great development of network techniques, especially wireless sensor networks [1], [2], [3].”
BackgroundTime-efficient power-aware scheduling for periodic real-time tasks
Chen, D., Hsu, C., & Lai, M.-F. (2009) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2009.5160972 · Cited by 4
Research works printed but not linked: 1
- huang, 2017, Hybrid DVFS scheduling for real-time systems based on reinforcement learning, Proc IEEE Int Conf Green Comput Commun Conf, 522
Cited by
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“computing resources of the edge device, in [257], DoubleDQL [89] is to determine the best Dynamic Voltage and Frequency Scaling (DVFS) algorithm.”
MethodA smart agriculture IoT system based on deep reinforcement learning
Bu, F., & Wang, X. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.04.041 · Cited by 342
Tuli, S., Ilager, S., Ramamohanarao, K., & Buyya, R. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.3017079 · Cited by 257
“Recent works have explored different value-based RL techniques to optimize several aspects of Resource Management Systems (RMS) in distributed environments [16], [17], [18, 19].”
BackgroundMethodEnergy-efficient edge based real-time healthcare support system
(2020) · Advances in Computers · doi:10.1016/bs.adcom.2019.09.007 · Cited by 211
AL-Jumaili, A. H. A., Muniyandi, R. C., Hasan, M. K., Paw, J. K. S., & Singh, M. S. J. (2023) · Sensors · doi:10.3390/s23062952 · Cited by 234
Metadata
- Authors
- Zhang, Q., Lin, M., Yang, L. T., Chen, Z., Khan, S. U., & Li, P.
- Affiliations
- University of Electronic Science and Technology of China (CN)
- St. Francis Xavier University (CA)
- Dalian University of Technology (CN)
- North Dakota State University (US)
- Venue
- IEEE Transactions on Services Computing
- Published
- 2019-09-01
- Volume
- 12
- Issue
- 5
- Pages
- 739-749
- ISSN
- 1939-1374 (ISSN-L)
- 2372-0204
- OpenAlex
- W2888896501
- Cited by
- 158
- References
- 33
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 16 · 2023: 20 · 2022: 21 · 2021: 29 · 2020: 29 · 2019: 22
- FWCI
- 12.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Services Computing (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Energy Efficiency in Computing
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- double deep Q-learning, edge computing, dynamic voltage and frequency scaling, target network, experience replay, rectified linear units, gradient vanishing, Q-function approximation, energy consumption reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.