Federated Deep Reinforcement Learning for Internet of Things With Decentralized Cooperative Edge Caching
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Edge caching is an emerging technology for addressing massive content access in mobile networks to support rapidly growing Internet-of-Things (IoT) services and applications. However, most current optimization-based methods lack a self-adaptive ability in dynamic environments. To tackle these challenges, current learning-based approaches are generally proposed in a centralized way. However, network resources may be overconsumed during the training and data transmission process. To address the complex and dynamic control issues, we propose a federated deep-reinforcement-learning-based cooperative edge caching (FADE) framework. FADE enables base stations (BSs) to cooperatively learn a shared predictive model by considering the first-round training parameters of the BSs as the initial input of the local training, and then uploads near-optimal local parameters to the BSs to participate in the next round of global training. Furthermore, we prove the expectation convergence of FADE. Trace-driven simulation results demonstrate the effectiveness of the proposed FADE framework on reducing the performance loss and average delay, offloading backhaul traffic, and improving the hit rate.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 1×
“We have the convergence in expectations: Thus, f ( w ) is proven to be bounded where for each UE j do Compute its local update.”
Federated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766 · cited 1×
“We have the convergence in expectations: Thus, f ( w ) is proven to be bounded where for each UE j do Compute its local update.”
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · cited 9×
“Caching replacement process of double DQN. Finally, we can obtain the Q -function iteration formula as where is the learning rate.”
InfluentialEfficient multi-user computation offloading for mobile-edge cloud computing
Chen, X., Jiao, L., Li, W., & Fu, X. (2016) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2015.2487344 · Cited by 2,675 · cited 1×
“Similar to [44], we can obtain the downlink data rate between BS n and UE u as follows:”
Adaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232 · cited 1×
“Similar to [51], we have the following assumptions.”
A Survey on Security and Privacy Issues in Internet-of-Things
Yang, Y., Wu, L., Yin, G., Li, L., & Zhao, H. (2017) · IEEE Internet of Things Journal · doi:10.1109/jiot.2017.2694844 · Cited by 1,278
FemtoCaching: Wireless Content Delivery Through Distributed Caching Helpers
Shanmugam, K., Golrezaei, N., Dimakis, A. G., Molisch, A. F., & Caire, G. (2013) · IEEE Transactions on Information Theory · doi:10.1109/tit.2013.2281606 · Cited by 1,246
Cache in the air: exploiting content caching and delivery techniques for 5G systems
Wang, X., Chen, M., Taleb, T., Ksentini, A., & Leung, V. (2014) · IEEE Communications Magazine · doi:10.1109/mcom.2014.6736753 · Cited by 1,063 · cited 2×
“The studies in [15] and [18]–[20] investigated the architectures of collaborative edge caching in mobile networks.”
Self-citationChen, X., Zhang, H., Wu, C., Mao, S., Ji, Y., & Bennis, M. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2876279 · Cited by 680
Communicating While Computing: Distributed mobile cloud computing over 5G heterogeneous networks
Barbarossa, S., Sardellitti, S., & Di Lorenzo, P. (2014) · IEEE Signal Processing Magazine · doi:10.1109/msp.2014.2334709 · Cited by 468
Big data caching for networking: moving from cloud to edge
Zeydan, E., Bastug, E., Bennis, M., Kader, M. A., Karatepe, I. A., Er, A. S., & Debbah, M. (2016) · IEEE Communications Magazine · doi:10.1109/mcom.2016.7565185 · Cited by 310 · cited 1×
“The studies in [15] and [18]–[20] investigated the architectures of collaborative edge caching in mobile networks.”
Learning-based optimization of cache content in a small cell base station
Blasco, P., & Gunduz, D. (2014) · 2014 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2014.6883600 · Cited by 298 · cited 2×
“placement [39] by using a multiarmed bandit (MAB) algorithm.”
Context-Aware Proactive Content Caching With Service Differentiation in Wireless Networks
Muller, S., Atan, O., van der Schaar, M., & Klein, A. (2017) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2016.2636139 · Cited by 281 · cited 4×
“Some state-of-the-art caching algorithms are shown as follows: including the LRU, LFU and FIFO, as well as Oracle [52], which is conducted as the baseline of best network performance.”
MethodInfluentialCooperative Edge Caching in User-Centric Clustered Mobile Networks
Zhang, S., He, P., Suto, K., Yang, P., Zhao, L., & Shen, X. (2018) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2017.2780834 · Cited by 279
Traffic Modeling and Proportional Partial Caching for Peer-to-Peer Systems
Hefeeda, M., & Saleh, O. (2008) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2008.918081 · Cited by 259
Wang, T., Zhang, G., Liu, A., Bhuiyan, M. Z. A., & Jin, Q. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2870288 · Cited by 253
A Survey on Mobile Data Offloading Technologies
Zhou, H., Wang, H., Li, X., & Leung, V. C. M. (2018) · IEEE Access · doi:10.1109/access.2018.2799546 · Cited by 171
Self-citationZeng, L., Li, E., Zhou, Z., & Chen, X. (2019) · IEEE Network · doi:10.1109/mnet.001.1800506 · Cited by 159
Deep Reinforcement Learning for Mobile Edge Caching: Review, New Features, and Open Issues
Zhu, H., Cao, Y., Wang, W., Jiang, T., & Jin, S. (2018) · IEEE Network · doi:10.1109/mnet.2018.1800109 · Cited by 153 · cited 1×
“Recently, learning-based approaches have also been widely utilized to design and optimize edge caching [27]–[30].”
Comparison of caching strategies in modern cellular backhaul networks
Woo, S., Jeong, E., Park, S., Lee, J., Ihm, S., & Park, K. (2013) · Proceeding of the 11th annual international conference on Mobile systems, applications, and services · doi:10.1145/2462456.2464442 · Cited by 150
Li, X., Wang, X., Wan, P.-J., Han, Z., & Leung, V. C. M. (2018) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2018.2844658 · Cited by 140
Self-citationMarket sharing games applied to content distribution in ad hoc networks
Goemans, M. X., Li, L., Mirrokni, V., & Thottan, M. (2006) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2006.872884 · Cited by 127
Collaborative Multi-Tier Caching in Heterogeneous Networks: Modeling, Analysis, and Design
Li, X., Wang, X., Li, K., Han, Z., & Leung, V. C. M. (2017) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2017.2734646 · Cited by 119 · cited 1×
“The studies in [16]–[19] investigated the architectures of collaborative edge caching in mobile networks.”
Self-citationPu, L., Chen, X., Mao, G., Xie, Q., & Xu, J. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2872436 · Cited by 115
Wang, X., Chen, M., Han, Z., Wu, D. O., & Kwon, T. T. (2014) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2014.6848179 · Cited by 110 · cited 1×
“, BSs) is similar to that in IoT devices, which improves the efficiency of content access compared to excessive downloading via backhaul links [15]–[17].”
Self-citationTraffic Offload Enhancements for eUTRAN
Samdanis, K., Taleb, T., & Schmid, S. (2011) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2011.072711.00168 · Cited by 107
Self-citationEdge-Caching Wireless Networks: Performance Analysis and Optimization
Vu, T. X., Chatzinotas, S., & Ottersten, B. (2018) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2018.2803816 · Cited by 101
In-network caching of Internet-of-Things data
Vural, S., Navaratnam, P., Wang, N., Wang, C., Dong, L., & Tafazolli, R. (2014) · 2014 IEEE International Conference on Communications (ICC) · doi:10.1109/icc.2014.6883811 · Cited by 99 · cited 1×
“[36] proposed a content replacement strategy with multiattribute joint optimization in terms of content lifetime, the request rate of users, and the hop information between the source user and the destination.”
Self-citationSaputra, Y. M., Hoang, D. T., Nguyen, D. N., Dutkiewicz, E., Niyato, D. T., & Kim, D. I. (2019) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2019.2912365 · Cited by 85
Computation Offloading with Multiple Agents in Edge-Computing–Supported IoT
Shen, S., Han, Y., Wang, X., & Wang, Y. (2020) · ACM Transactions on Sensor Networks · doi:10.1145/3372025 · Cited by 85
Self-citationDistributed cache replacement for caching-enable base stations in cellular networks
Gu, J., Wang, W., Huang, A., Shan, H., & Zhang, Z. (2014) · International Conference on Communications · doi:10.1109/icc.2014.6883723 · Cited by 78
Delay performance analysis of cooperative cell caching in future mobile networks
Li, X., Wang, X., Xiao, S., & Leung, V. C. M. (2015) · International Conference on Communications · doi:10.1109/icc.2015.7249223 · Cited by 72 · cited 1×
“[23], and Chae et al.”
Self-citationToward Smart and Cooperative Edge Caching for 5G Networks: A Deep Learning Based Approach
Pang, H., Liu, J., Fan, X., & Sun, L. (2018) · International Workshop on Quality of Service · doi:10.1109/iwqos.2018.8624176 · Cited by 66 · cited 1×
“Recently, learning-based approaches have also been widely utilized to design and optimize edge caching [27]–[30].”
Sheng, M., Xu, C., Liu, J., Song, J., Ma, X., & Li, J. (2016) · IEEE Communications Magazine · doi:10.1109/mcom.2016.7537179 · Cited by 62
Collaborative Edge Caching in Context-Aware Device-to-Device Networks
Zhao, X., Yuan, P., li, H., & Tang, S. (2018) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2018.2858254 · Cited by 55
A novel caching scheme for Internet based mobile ad hoc networks
Lim, S., Lee, W.-C., Cao, G., & Das, C. R. (2004) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn.2003.1284147 · Cited by 54
Cooperative Transmission via Caching Helpers
Chae, S. H., Ryu, J. Y., Quek, T. Q. S., & Choi, W. (2015) · Globecom · doi:10.1109/glocom.2015.7417164 · Cited by 54
Optimal Probabilistic Caching in Heterogeneous IoT Networks
Zhang, S., & Liu, J. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2969466 · Cited by 53
Cache-Aware Computation Offloading in IoT Systems
Wei, H., Luo, H., Sun, Y., & Obaidat, M. S. (2019) · IEEE Systems Journal · doi:10.1109/jsyst.2019.2903293 · Cited by 50
Wu, Q., Chen, X., Zhou, Z., & Chen, L. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2903120 · Cited by 41
Cooperative Transmission via Caching Helpers
Chae, S. H., Ryu, J. Y., Quek, T. Q. S., & Choi, W. (2014) · Globecom · doi:10.1109/glocom.2014.7417164 · Cited by 39
IoTCache: Toward Data-Driven Network Caching for Internet of Things
Chen, B., Liu, L., Sun, M., & Ma, H. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2935442 · Cited by 37
Deep Reinforcement Learning for Cooperative Edge Caching in Future Mobile Networks
Ding, L., Han, Y., Wang, C., Shi, G., Wang, X., Li, X., & Leung, V. C. M. (2019) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc.2019.8885516 · Cited by 33 · cited 1×
“Facing the tremendous growth in network traffic load and Quality of Service/Experience (QoS/QoE) of user demands, enormous challenges have emerged for mobile networks and the IoT [2]–[7].”
Self-citationCollaborative Edge Caching through Service Function Chaining: Architecture and Challenges
Lei, L., Xiong, X., Hou, L., & Zheng, K. (2018) · IEEE Wireless Communications · doi:10.1109/mwc.2018.1700321 · Cited by 32
Zhang, M., Chen, J., He, S., Yang, L., Gong, X., & Zhang, J. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2938491 · Cited by 31 · cited 1×
“In addition, most of the traditional DRL algorithms train the data in BSs or a datacenter by sharing the original data, leading to a large amount of network resource consumption during the data transmission process [31], [32].”
Random Caching Optimization in Large-Scale Cache-Enabled Internet of Things Networks
Han, Y., Wang, R., & Wu, J. H. (2019) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2019.2894033 · Cited by 23
How to Cache in Mobile Hybrid IoT Networks?
Do, T.-A., Jeon, S.-W., & Shin, W.-Y. (2019) · IEEE Access · doi:10.1109/access.2018.2882074 · Cited by 15
Dai, M., Deng, H., Chen, B., Su, G., Lin, X., & Wang, H. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2952643 · Cited by 13
Survey on Cache Schemes in Heterogeneous Networks using 5G Internet of Things
Ajmal, S., Muzammil, M. B., Jamil, A., Abbas, S. M., Iqbal, U., & Touseef, P. (2019) · International Conference on Future Networks and Distributed Systems (ICFNDS) · doi:10.1145/3341325.3342027 · Cited by 9
Edge-Caching Wireless Networks: Energy-Efficient Design and Optimization.Preprint
Vu, T. X., Chatzinotas, S., & Ottersten, B. (2017) · arXiv (Cornell University) · Cited by 5
Exploiting the edge power: an edge deep learning framework
Fan, X., Huang, Y., Ma, X., Liu, J., & Leung, V. C. M. (2018) · CCF Transactions on Networking · doi:10.1007/s42045-018-0003-0 · Cited by 5
Self-citation
Research works printed but not linked: 1
- 2020, Apple Edge Cache
Cited by
Federated Learning for Internet of Things: A Comprehensive Survey
Nguyen, D. C., Ding, M., Pathirana, P. N., Seneviratne, A., Li, J., & Vincent Poor, H. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3075439 · Cited by 1,503
A Survey on Security and Privacy Issues in Edge-Computing-Assisted Internet of Things
Alwarafy, A., Al-Thelaya, K. A., Abdallah, M., Schneider, J., & Hamdi, M. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3015432 · Cited by 411
“Hence, achieving secure and privacy-preserving data analysis in the EC-assisted paradigm based on distributed/federated learning strategies [102] without the leakage of the private training dataset is still an open research direction.”
BackgroundFederated Learning for 6G: Applications, Challenges, and Opportunities
Yang, Z., Chen, M., Wong, K.-K., Poor, H. V., & Cui, S. (2022) · Engineering · doi:10.1016/j.eng.2021.12.002 · Cited by 374
Yu, S., Chen, X., Zhou, Z., Gong, X., & Wu, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3026589 · Cited by 353
“e the formulated optimization problem, and present a novel two-timescale deep reinforcement learning (2Ts-DRL) approach to train the decision agent, as shown in Fig. 5. In reinforcement learning (RL) [9], an agent can periodically learn to take actions, observes the most reward and automatically adjusts its strategy i…”
BackgroundMethodInfluentialFusion of Federated Learning and Industrial Internet of Things: A survey
Boobalan, P., Ramu, S. P., Pham, Q.-V., Dev, K., Pandya, S., Maddikunta, P. K. R., Gadekallu, T. R., & Huynh-The, T. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109048 · Cited by 280
Metadata
- Authors
- Wang, X., Wang, C., Li, X., Leung, V. C. M., & Taleb, T.
- Affiliations
- Tianjin University (CN)
- Chongqing University (CN)
- University of British Columbia (CA)
- Sejong University (KR)
- Venue
- IEEE Internet of Things Journal
- Published
- 2020-10
- Volume
- 7
- Issue
- 10
- Pages
- 9441-9455
- ISSN
- 2327-4662 (ISSN-L)
- 2372-2541
- OpenAlex
- W3016207298
- Cited by
- 402
- References
- 51
- Citations by year
- 2026: 12 · 2025: 52 · 2024: 79 · 2023: 78 · 2022: 74 · 2021: 74 · 2020: 33
- FWCI
- 30.43
- Type
- journal article
- Language
- en
- Open access
- green
- research.aalto.fi
- Licence
- Locations
- IEEE Internet of Things Journal (publishedVersion)
- Aaltodoc (Aalto University) (submittedVersion, other-oa)
- Funders
- National Key Research and Development Program of China (2019YFB2101901, 2018YFC0809803, 2018YFF0214700, 2018YFB2100100)
- China NSFC (61702364, 61902044)
- Chongqing Research Program of Basic Research and Frontier Technology (cstc2019jcyj-msxmX0589)
- Chinese National Engineering Laboratory for Big Data System Computing Technology
- Canadian NSERC
- European Unions Horizon 2020 Research and Innovation Program through the MonB5G Project (871780)
- Academy of Finland 6Genesis Project (318927)
- Academy of Finland CSN Project (311654)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Caching and Content Delivery
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Computer Networks and Communications › Age of Information Optimization
- Keywords
- federated deep reinforcement learning, edge caching, Internet of Things, base station cooperation, content prefetching, hit rate, mobile edge computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.