Distributed Q-Learning Aided Uplink Grant-Free NOMA for Massive Machine-Type Communications
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
The explosive growth of machine-type communications (MTC) devices poses critical challenges to the existing cellular networks. Therefore, how to support massive MTC devices with limited resources is an urgent problem to be solved. Bursty traffic is an important characteristic of MTC devices, which makes it difficult for agents to learn useful experience and has a negative impact on model convergence. However, most existing reinforcement learning-based literatures assume that devices have saturate data. Towards this end, we propose two distributed Q-learning aided uplink grant-free non-orthogonal multiple access (NOMA) schemes (including all-devices distributed Q-learning (ADDQ) scheme and portion-devices distributed Q-learning (PDDQ) scheme) to maximize the number of accessible devices, where the bursty traffic of massive MTC devices is carefully considered. In order to reduce the dimension of scheduling space and mitigate the impact of bursty traffic, the idea of grouping devices as well as transmission resources and the intermittent learning mode are adopted in our schemes. Extensive numerical results demonstrate the advantages of proposed schemes from multiple perspectives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Edge AI can also help design AI-native communication strategies for the physical layer (e.g., task-oriented semantic communication [58]) and medium access control layer (e.g., random access protocol [59]).”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [59]
Citations
Cites
Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y.-C., & Kim, D. I. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916583 · Cited by 2,060
Deep Reinforcement Learning Based Resource Allocation for V2V Communications
Ye, H., Li, G. Y., & Juang, B.-H. F. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2897134 · Cited by 870
Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning
Liang, L., Ye, H., & Li, G. Y. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2933962 · Cited by 559 · cited 1×
“This fits well with the nature of RL [23].”
BackgroundDeep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access
Naparstek, O., & Cohen, K. (2019) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2018.2879433 · Cited by 477 · cited 3×
“Therefore, the traditional RL-assisted schemes [14]–[17], in which the agent learns in each time slot, are difficult to converge in the mMTC scenario considering the devices’ bursty traffic.”
BackgroundMethodGrant-Free Non-Orthogonal Multiple Access for IoT: A Survey
Shahab, M. B., Abbas, R., Shirvanimoghaddam, M., & Johnson, S. J. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2996032 · Cited by 453 · cited 2×
“One is that the BS pre-allocates radio resources for each device, and the other is that the device chooses transmission resources by itself [1].”
BackgroundSharma, S. K., & Wang, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916177 · Cited by 427 · cited 1×
“Without loss of generality, it is assumed that the active probability of each device follows uniform distribution [19] and that P a = { p a 1 , p a 2 , · · · , p aM } is the set of all devices’ active probability.”
BackgroundGhavimi, F., & Chen, H.-H. (2015) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2361626 · Cited by 348 · cited 1×
“In the current LTE/LTE-A system, an active device first requests transmission resources from base station (BS) in a grant-based manner for subsequent data transmission [3], [4].”
BackgroundUplink contention based SCMA for 5G radio access
Au, K., Zhang, L., Nikopour, H., Yi, E., Bayesteh, A., Vilaipornsawai, U., Ma, J., & Zhu, P. (2014) · 2014 IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/glocomw.2014.7063547 · Cited by 289 · cited 2×
“Similar to the existing literatures [9]–[11], [14], [20], we ignore the possibility that some signals in collision can be decoded successfully due to capture effects [9], [22].”
ResultNOMA-Based Random Access With Multichannel ALOHA
Choi, J. (2017) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2017.2766778 · Cited by 276 · cited 3×
“Similar to the existing literatures [9]–[11], [14], [20], we ignore the possibility that some signals in collision can be decoded successfully due to capture effects [9], [22].”
MethodResultBlind detection of SCMA for uplink grant-free multiple-access
Bayesteh, A., Yi, E., Nikopour, H., & Baligh, H. (2014) · 2014 11th International Symposium on Wireless Communications Systems (ISWCS) · doi:10.1109/iswcs.2014.6933472 · Cited by 171 · cited 1×
“Although it is unrealistic for a BS to estimate the channel condition for massive devices in the mMTC scenario, some blind detection and compressive sensing techniques can help BS estimate channel condition by narrowing the range of devices [21].”
LTE/LTE-A Random Access for Massive Machine-Type Communications in Smart Cities
Ali, M. S., Hossain, E., & Kim, D. I. (2017) · IEEE Communications Magazine · doi:10.1109/mcom.2017.1600215cm · Cited by 167 · cited 1×
“A collision-resolution-based access model was proposed in [8], which reduced the collision probability by reserving a preamble set for devices that had already collided.”
MethodDu, Y., Dong, B., Chen, Z., Wang, X., Liu, Z., Gao, P., & Li, S. (2017) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2017.2726279 · Cited by 159 · cited 1×
“Among them, sporadic transmission means that MTC devices are in the state of no data transmission most of the time and the number of active devices in the same time slot is only a small part of all devices [2].”
BackgroundNon-Orthogonal Random Access for 5G Networks
Liang, Y., Li, X., Zhang, J., & Ding, Z. (2017) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2017.2703168 · Cited by 148 · cited 2×
“In the current LTE/LTE-A system, an active device first requests transmission resources from base station (BS) in a grant-based manner for subsequent data transmission [3], [4].”
BackgroundNOMA-Assisted Machine-Type Communications in UDN: State-of-the-Art and Challenges
Elbayoumi, M., Kamel, M., Hamouda, W., & Youssef, A. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2977845 · Cited by 146 · cited 1×
“A large number of access requests at the same time can cause severe preamble collisions, resulting in serious transmission delay and network congestion [5].”
BackgroundDeep Reinforcement Learning for Throughput Improvement of the Uplink Grant-Free NOMA System
Zhang, J., Tao, X., Wu, H., Zhang, N., & Zhang, X. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2972274 · Cited by 133 · cited 2×
“As the power-domain NOMA was considered in literature [16], it tried to assign devices with a large difference in channel conditions to a group as far as possible.”
MethodOn the Fundamental Limits of Random Non-Orthogonal Multiple Access in Cellular Massive IoT
Shirvanimoghaddam, M., Condoluci, M., Dohler, M., & Johnson, S. J. (2017) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2017.2724442 · Cited by 131 · cited 2×
“The authors in [10] proposed a grant-free random NOMA strategy in which active devices transmitted data over a randomly selected available channel.”
BackgroundMethodA Novel Analytical Framework for Massive Grant-Free NOMA
Abbas, R., Shirvanimoghaddam, M., Li, Y., & Vucetic, B. (2019) · IEEE Transactions on Communications · doi:10.1109/tcomm.2018.2881120 · Cited by 126 · cited 2×
“Similar to the existing literatures [9]–[11], [14], [20], we ignore the possibility that some signals in collision can be decoded successfully due to capture effects [9], [22].”
ResultReinforcement Learning for Real-Time Optimization in NB-IoT Networks
Jiang, N., Deng, Y., Nallanathan, A., & Chambers, J. A. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904366 · Cited by 116 · cited 1×
“In particular, the authors in [18] introduced RL at the BS side to maximize the long-term average number of served devices.”
BackgroundOn the fundamental limits of random non-orthogonal multiple access in cellular massive IoT
Shirvanimoghaddam, M., Condoluci, M., Döhler, M., & Johnson, S. J. (2017) · Figshare · Cited by 92
Collaborative Distributed Q-Learning for RACH Congestion Minimization in Cellular IoT Networks
Sharma, S. K., & Wang, X. (2019) · IEEE Communications Letters · doi:10.1109/lcomm.2019.2896929 · Cited by 88 · cited 6×
“It is worth noting that all of the above RL-assisted schemes [14]–[17] assume that the device has saturated traffic (i.e., the device essentially always has data to be transmitted).”
BackgroundMethodResultInfluentialA NOMA-Based Q-Learning Random Access Method for Machine Type Communications
da Silva, M. V., Souza, R. D., Alves, H., & Abrão, T. (2020) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2020.3002691 · Cited by 74 · cited 1×
“On the basis of [14], the authors in [15] introduced power-domain NOMA to further improve network throughput and defined a new reward that enabled only one acknowledgement bit returning to the device from the BS in each time slot.”
BackgroundThe effect of capture on performance of multichannel slotted ALOHA systems
Yue, W. (1991) · IEEE Transactions on Communications · doi:10.1109/26.87169 · Cited by 64 · cited 1×
“Similar to the existing literatures [9]–[11], [14], [20], we ignore the possibility that some signals in collision can be decoded successfully due to capture effects [9], [22].”
ResultSARA: Sparse Code Multiple Access-Applied Random Access for IoT Devices
Moon, S., Lee, H.-S., & Lee, J.-W. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2835828 · Cited by 53 · cited 1×
“As for grant-based access schemes, the authors in [7] proposed a random access framework based on code-domain NOMA.”
BackgroundCoded Tandem Spreading Multiple Access for Massive Machine-Type Communications
Ma, G., Ai, B., Wang, F., Chen, X., Zhong, Z. D., Zhao, Z., & Guan, H. (2018) · IEEE Wireless Communications · doi:10.1109/mwc.2018.1700107 · Cited by 43 · cited 1×
“The authors of [12] proposed a coded tandem spreading multiple access method, which could reduce the probability of packet collisions by adding redundant segments into packets and using multiple orthogonal spreading codes to connect different segments.”
BackgroundApplications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y.-C., & Kim, D. I. (2018) · arXiv · doi:10.48550/arxiv.1810.07862 · Cited by 17
A Novel Perspective on Multiple Access in 5G Network: Framework and Solutions
Peng, W., Gao, W., & Liu, J. (2019) · IEEE Wireless Communications · doi:10.1109/mwc.2019.1800315 · Cited by 16 · cited 1×
“Among them, since non-orthogonal multiple access (NOMA) allows multiple devices to share the same time-frequency resource block via power domain or code domain multiplexing, it has the inherent advantage of large-scale device access compared to orthogonal multiple access [6].”
BackgroundSelf-citation
Research works printed but not linked: 1
- azar, 2011, Speedy Q-learning, Adv Neural Inf Process Syst, 24, 2411
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Edge AI can also help design AI-native communication strategies for the physical layer (e.g., task-oriented semantic communication [58]) and medium access control layer (e.g., random access protocol [59]).”
BackgroundJoint Activity and Channel Estimation for Extra-Large MIMO Systems
Iimori, H., Takahashi, T., Ishibashi, K., de Abreu, G. T. F., David González, G., & Gonsa, O. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3157271 · Cited by 63
Optimization of Grant-Free NOMA With Multiple Configured-Grants for mURLLC
Yan, L., Deng, Y., Elkashlan, M., Nallanathan, A., & Karagiannidis, G. K. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3143264 · Cited by 57
Optimal Beamformer Design for Millimeter Wave Dual-Functional Radar-Communication Based V2X Systems
Liu, B., Liu, J., & Kato, N. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3196089 · Cited by 52
Self-citationNon-Orthogonal Multiple Access Assisted Secure Computation Offloading via Cooperative Jamming
Wu, Y., Ji, G., Wang, T., Qian, L., Lin, B., & Shen, X. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3167861 · Cited by 51
Metadata
- Authors
- Liu, J., Shi, Z., Zhang, S., & Kato, N.
- Affiliations
- Northwestern Polytechnical University (CN)
- Xidian University (CN)
- Tohoku University (JP)
- Venue
- IEEE Journal on Selected Areas in Communications
- Published
- 2021-07
- Volume
- 39
- Issue
- 7
- Pages
- 2029-2041
- ISSN
- 0733-8716 (ISSN-L)
- 1558-0008
- OpenAlex
- W3160612200
- Cited by
- 53
- References
- 25
- Citations by year
- 2026: 5 · 2025: 3 · 2024: 8 · 2023: 14 · 2022: 18 · 2021: 5
- FWCI
- 3.31
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Journal on Selected Areas in Communications (publishedVersion)
- Funders
- National Natural Science Foundation of China (61771374, 61771373, 61801360, 62001393)
- Natural Science Basic Research Program of Shaanxi (2020JC-15, 2020JM-109)
- Fundamental Research Funds for the Central Universities (31020200QD010)
- Xi’an Unmanned System Security and Intelligent Communications ISTC Center
- Special Funds for Central Universities Construction of World-Class Universities (Disciplines) and Special Development Guidance (0639021GH0201024)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › IoT Networks and Protocols
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Wireless Communication Technologies
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Keywords
- distributed Q-learning, grant-free access, non-orthogonal multiple access, uplink transmission, massive machine-type communications, bursty traffic, device grouping, resource scheduling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.