The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-Time Popularities
Open accessAbstract
Small basestations (SBs) equipped with caching units have potential to handle the unprecedented demand growth in heterogeneous networks. Through low-rate, backhaul connections with the backbone, SBs can prefetch popular files during off-peak traffic hours, and service them to the edge at peak periods. To intelligently prefetch, each SB must learn what and when to cache, while taking into account SB memory limitations, the massive number of available contents, the unknown popularity profiles, as well as the space-time popularity dynamics of user file requests. In this paper, local and global Markov processes model user requests, and a reinforcement learning (RL) framework is put forth for finding the optimal caching policy when the transition probabilities involved are unknown. Joint consideration of global and local popularity demands along with cache-refreshing costs allow for a simple, yet practical asynchronous caching approach. The novel RL-based caching relies on a Q-learning algorithm to implement the optimal policy in an online fashion, thus, enabling the cache control unit at the SB to learn, track, and possibly adapt to the underlying dynamics. To endow the algorithm with scalability, a linear function approximation of the proposed Q-learning scheme is introduced, offering faster convergence as well as reduced complexity and memory requirements. Numerical tests corroborate the merits of the proposed approach in various realistic settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“But [7] is based on Q-Learning, which is not feasible for the practical MEC system where the stateaction space is tremendous”
From In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning (2019) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,191
Watkins, C., & Dayan, P. (1992) · Machine-mediated learning · doi:10.1023/a:1022676722315 · Cited by 14,277
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,730
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,825
Andrews, J. G., Buzzi, S., Choi, W., Hanly, S. V., Lozano, A., Soong, A. C. K., & Zhang, J. C. (2014) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2014.2328098 · Cited by 8,106
Web caching and Zipf-like distributions: evidence and implications
Breslau, L., Cao, P., Fan, L., Phillips, G., & Shenker, S. J. (1999) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.1999.749260 · Cited by 3,485
Device-to-device communication as an underlay to LTE-advanced networks
Doppler, K., Rinne, M., Wijting, C., Ribeiro, C. B., & Hugl, K. (2009) · IEEE Communications Magazine · doi:10.1109/mcom.2009.5350367 · Cited by 1,904
Maddah-Ali, M. A., & Niesen, U. (2014) · IEEE Transactions on Information Theory · doi:10.1109/tit.2014.2306938 · Cited by 1,726
Femtocells: Past, Present, and Future
Andrews, J. G., Claußen, H., Döhler, M., Rangan, S., & Reed, M. C. (2012) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2012.120401 · Cited by 1,184
Cache in the air: exploiting content caching and delivery techniques for 5G systems
Wang, X., Chen, M., Taleb, T., Ksentini, A., & Leung, V. (2014) · IEEE Communications Magazine · doi:10.1109/mcom.2014.6736753 · Cited by 1,063 · cited 1×
“, reusable video streams, during off-peak periods, storing this data in SBs equipped with memory units, and reusing them during peak traffic hours [2]–[4].”
MethodFemtocaching and device-to-device collaboration: A new architecture for wireless video distribution
Golrezaei, N., Molisch, A. F., Dimakis, A. G., & Caire, G. (2013) · IEEE Communications Magazine · doi:10.1109/mcom.2013.6495773 · Cited by 686
Decentralized Coded Caching Attains Order-Optimal Memory-Rate Tradeoff
Maddah-Ali, M. A., & Niesen, U. (2014) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2014.2317316 · Cited by 667
Fundamental Limits of Caching in Wireless D2D Networks
Ji, M., Caire, G., & Molisch, A. F. (2015) · IEEE Transactions on Information Theory · doi:10.1109/tit.2015.2504556 · Cited by 635
The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
Borkar, V. S., & Meyn, S. (2000) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012997331639 · Cited by 528
Wireless Device-to-Device Caching Networks: Basic Principles and System Performance
Ji, M., Caire, G., & Molisch, A. F. (2015) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2015.2452672 · Cited by 445
Optimal geographic caching in cellular networks
Błaszczyszyn, B., & Giovanidis, A. (2015) · International Conference on Communications · doi:10.1109/icc.2015.7248843 · Cited by 428
Kernel-Based Reinforcement Learning
Ormoneit, D., & Sen, S. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1_100235 · Cited by 376
Wireless caching: technical misconceptions and business barriers
Paschos, G. S., Baştuǧ, E., Land, I., Caire, G., & Debbah, M. A. (2016) · IEEE Communications Magazine · doi:10.1109/mcom.2016.7537172 · Cited by 321
Context-Aware Proactive Content Caching With Service Differentiation in Wireless Networks
Muller, S., Atan, O., van der Schaar, M., & Klein, A. (2016) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2016.2636139 · Cited by 282
Kernel-Based Reinforcement Learning
Ormoneit, D., & Sen, Ś. (2002) · Machine Learning · doi:10.1023/a:1017928328829 · Cited by 265
Pedarsani, R., Maddah-Ali, M. A., & Niesen, U. (2015) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2015.2394482 · Cited by 236
A Learning-Based Approach to Caching in Heterogenous Small Cell Networks
Bharath, B. N., Nagananda, K. G., & Poor, H. V. (2016) · IEEE Transactions on Communications · doi:10.1109/tcomm.2016.2536728 · Cited by 202
Placing dynamic content in caches with small population
Leconte, M., Paschos, G. S., Gkatzikis, L., Draief, M., Vassilaras, S., & Chouvardas, S. (2016) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2016.7524380 · Cited by 155
Learning distributed caching strategies in small cell networks
Sengupta, A., Amuru, S., Tandon, R., Buehrer, R. M., & Clancy, T. C. (2014) · International Symposium on Wireless Communication Systems · doi:10.1109/iswcs.2014.6933484 · Cited by 141
A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning
Geramifard, A., Walsh, T. J., Tellex, S., Chowdhary, G. V., Roy, N., & How, J. P. (2013) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000042 · Cited by 124
Video delivery over heterogeneous cellular networks: Optimizing cost and performance
Poularakis, K., Iosifidis, G., Argyriou, A., & Tassiulas, L. (2014) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2014.6848038 · Cited by 112
Trend-Aware Video Caching Through Online Learning
Li, S., Xu, J., van der Schaar, M., & Li, W. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2596042 · Cited by 98
Decentralized coded caching attains order-optimal memory-rate tradeoff
Maddah-Ali, M. A., & Niesen, U. (2013) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2013.6736555 · Cited by 95
A transfer learning approach for cache-enabled wireless networks
Bastug, E., Bennis, M., & Debbah, M. A. (2015) · International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) · doi:10.1109/wiopt.2015.7151068 · Cited by 94
Optimal Caching and Scheduling for Cache-Enabled D2D Communications
Chen, B., Yang, C., & Xiong, Z. (2017) · IEEE Communications Letters · doi:10.1109/lcomm.2017.2652440 · Cited by 82
Exploration and Inference in Learning from Reinforcement
Wyatt, J. (1998) · Cited by 74
Content caching at the wireless network edge: A distributed algorithm via belief propagation
Liu, J., Bai, B. B., Zhang, J., & Letaief, K. B. (2016) · International Conference on Communications · doi:10.1109/icc.2016.7510807 · Cited by 61
Learning Representation and Control in Markov Decision Processes: New Frontiers
Sridhar, M. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000003 · Cited by 55
Ultra-dense edge caching under spatio-temporal demand and network dynamics
Kim, H., Park, J., Bennis, M., Kim, S., & Debbah, M. A. (2017) · International Conference on Communications · doi:10.1109/icc.2017.7996764 · Cited by 25
A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning
Geramifard, A. (2013) · now publishers, Inc. eBooks · doi:10.1561/9781601987617 · Cited by 19
Learning Representation and Control in Markov Decision Processes: New Frontiers
Mahadevan, S. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601982391 · Cited by 17
Blasco, P., & Gündüz, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.6154 · Cited by 15
Ultra-Dense Edge Caching under Spatio-Temporal Demand and Network DynamicsPreprint
Kim, H., Park, J., Bennis, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01038 · Cited by 1
Research works printed but not linked: 1
- lillicrap, 2016, Proc Int Conf Learn Representations
Cited by
Sorted by citations
In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
Wang, X., Han, Y., Wang, C., Zhao, Q., Chen, X., & Chen, M. (2019) · IEEE Network · doi:10.1109/mnet.2019.1800286 · Cited by 1,030
“Particularly, related studies on edge computing and caching, such as [7] [8], have shown that reinforcement”
BackgroundMethodMorocho-Cayamcela, M. E., Lee, H., & Lim, W. (2019) · IEEE Access · doi:10.1109/access.2019.2942390 · Cited by 442
Deep Reinforcement Learning for Cooperative Content Caching in Vehicular Edge Computing and Networks
Qiao, G., Leng, S., Maharjan, S., Zhang, Y., & Ansari, N. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2945640 · Cited by 321
The Role of Caching in Future Communication Systems and Networks
Paschos, G. S., Iosifidis, G., Tao, M., Towsley, D., & Caire, G. (2018) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2018.2844939 · Cited by 259
Online Proactive Caching in Mobile Edge Computing Using Bidirectional Deep Recurrent Neural Network
Ale, L., Zhang, N., Wu, H., Chen, D., & Han, T. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2903245 · Cited by 187
Metadata
- Authors
- Sadeghi, A., Sheikholeslami, F., & Giannakis, G. B.
- Affiliations
- University of Minnesota (US)
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2018-02
- Volume
- 12
- Issue
- 1
- Pages
- 180-190
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W2746390786
- Cited by
- 257
- References
- 39
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 23 · 2023: 21 · 2022: 32 · 2021: 41 · 2020: 57 · 2019: 58 · 2018: 15
- FWCI
- 23.68
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1423316, 1508993, 1514056, 1711471)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Caching and Content Delivery
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Computer Networks and Communications › Age of Information Optimization
- Keywords
- content caching, small cells, 5G, reinforcement learning, Q-learning, prefetching, edge caching, Markov processes, linear function approximation, scalability, heterogeneous networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
