Joint Deep Reinforcement Learning and Unfolding: Beam Selection and Precoding for mmWave Multiuser MIMO With Lens Arrays
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
The millimeter wave (mmWave) multiuser multiple-input multiple-output (MU-MIMO) systems with discrete lens arrays (DLA) have received great attention due to their simple hardware implementation and excellent performance. In this work, we investigate the joint design of beam selection and digital precoding matrices for mmWave MU-MIMO systems with DLA to maximize the sum-rate subject to the transmit power constraint and the constraints of the selection matrix structure. The investigated non-convex problem with discrete variables and coupled constraints is challenging to solve and an efficient framework of joint neural network (NN) design is proposed to tackle it. Specifically, the proposed framework consists of a deep reinforcement learning (DRL)-based NN and a deep-unfolding NN, which are employed to optimize the beam selection and digital precoding matrices, respectively. As for the DRL-based NN, we formulate the beam selection problem as a Markov decision process and a double deep Q-network algorithm is developed to solve it. The base station is considered to be an agent, where the state, action, and reward function are carefully designed. Regarding the design of the digital precoding matrix, we develop an iterative weighted minimum mean-square error algorithm induced deep-unfolding NN, which unfolds this algorithm into a layer-wise structure with introduced trainable parameters. Simulation results verify that this jointly trained NN remarkably outperforms the existing itera…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To reduce the model and sample complexity, as well as improve the performance and interpretability, unfolded neural networks were developed in [267]- [269] to parameterize the iterative policy via unfolding one iteration of the existing structured algorithm into one layer of a neural network.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [268]
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“In addition, deep reinforcement learning (DRL) [27] has also been widely applied to solve the high-dimensional non-convex optimization problems.”
MethodNoncooperative Cellular Wireless with Unlimited Numbers of Base Station Antennas
Marzetta, T. L. (2010) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2010.092810.091092 · Cited by 6,302 · cited 1×
“3087233 of antennas at the transceiver in a cellular network, where the massive multiple-input multiple-output (MIMO) techniques can be readily realized to perform highly directional transmissions [2]–[4].”
BackgroundScaling Up MIMO: Opportunities and Challenges with Very Large Arrays
Rusek, F., Persson, D., Lau, B. K., Larsson, E. G., Marzetta, T. L., & Tufvesson, F. (2013) · IEEE Signal Processing Magazine · doi:10.1109/msp.2011.2178495 · Cited by 4,654 · cited 1×
“3087233 of antennas at the transceiver in a cellular network, where the massive multiple-input multiple-output (MIMO) techniques can be readily realized to perform highly directional transmissions [2]–[4].”
BackgroundScaling up MIMO: Opportunities and challenges with very large arrays
Rusek, F., Persson, D., Lau, B. K., Larsson, E. G., Marzetta, T. L., Edfors, O., Tufvesson, F., Print, P., Rusek, F., Persson, D., et al. (2013) · Cited by 4,089
Five disruptive technology directions for 5G
Boccardi, F., Heath, R. W., Lozano, A., Marzetta, T. L., & Popovski, P. (2014) · IEEE Communications Magazine · doi:10.1109/mcom.2014.6736746 · Cited by 3,709
Spatially Sparse Precoding in Millimeter Wave MIMO Systems
Ayach, O. E., Rajagopal, S., Abu-Surra, S., Pi, Z., & Heath, R. W. (2014) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2014.011714.130846 · Cited by 3,557 · cited 1×
“The hybrid analog-digital (AD) transceiver consisting of cascaded baseband digital precoder and RF analog beamformer has been proposed as a trade-off between cost and performance, which employs less RF chains than the number of antennas, while imposing a constant modulus constraint on the analog beamforming matrix [5].”
BackgroundDeep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · cited 2×
“We apply the DDQN [36] to learn the target value y t to avoid the overestimation and make the NN stable, which is expressed as where θ (cid:48) and θ denote the parameters of the two NNs, which have been applied to select and evaluate the action, respectively.”
MethodShi, Q., Razaviyayn, M., Luo, Z.-Q., & He, C. (2011) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2011.2147784 · Cited by 2,129 · cited 5×
“Given the beam selection matrix, the problem of the digital precoding design is a classic sum-rate maximization problem with power constraints that can be settled by the iterative WMMSE algorithm [36].”
BackgroundMethodInfluentialHuang, L., Bi, S., & Zhang, Y.-J. A. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2928811 · Cited by 1,040 · cited 5×
“The DQN has been widely applied in mobile offloading [28], dynamic channel access [29], radio access [30], resource allocation [31], [32], and mobile-edge computing (MEC) [33].”
BackgroundMethodInfluentialLearning to Optimize: Training Deep Neural Networks for Interference Management
Sun, H., Chen, X., Shi, Q., Hong, M., Fu, X., & Sidiropoulos, N. D. (2018) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2866382 · Cited by 874 · cited 2×
“Recently, the machine learning techniques emerge as an efficient method and there are a number of studies employing the neural networks (NNs) for solving the investigated problems [16], such as precoding design [17], beam selection [18]–[20], and resource allocation [21].”
BackgroundMethodShi, Q., Razaviyayn, M., Luo, Z.-Q., & He, C. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5946304 · Cited by 873
Millimeter-Wave Enhanced Local Area Systems: A High-Data-Rate Approach for Future Wireless Networks
Ghosh, A., Thomas, T. A., Cudak, M. C., Ratasuk, R., Moorut, P., Vook, F. W., Rappaport, T. S., MacCartney, G. R., Sun, S., & Nie, S. (2014) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2014.2328111 · Cited by 796
Brady, J., Behdad, N., & Sayeed, A. M. (2013) · IEEE Transactions on Antennas and Propagation · doi:10.1109/tap.2013.2254442 · Cited by 731 · cited 1×
“To address these issues, a promising mmWave MIMO technique with discrete lens arrays (DLA) has been proposed in [6] to reduce hardware implementation complexity without remarkable performance loss [7]–[15].”
MethodDeep Learning in Physical Layer Communications
Qin, Z., Ye, H., Li, G. Y., & Juang, B.-H. F. (2019) · IEEE Wireless Communications · doi:10.1109/mwc.2019.1800601 · Cited by 692
Deep Learning in Physical Layer Communications
Qin, Z., Ye, H., Li, G. Y., & Juang, B.-H. F. (2018) · arXiv · doi:10.48550/arxiv.1807.11713 · Cited by 692
Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks
Cui, J., Liu, Y., & Nallanathan, A. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2935201 · Cited by 593 · cited 1×
“The DQN has been widely applied in mobile offloading [28], dynamic channel access [29], radio access [30], unmanned aerial vehicle (UAV) [31], intelligent reflecting surface (IRS) [32], resource allocation [33], [34], and mobile-edge computing (MEC) [35].”
BackgroundSelf-citationDeep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks
Wang, S., Liu, H., Gomes, P. H., & Krishnamachari, B. (2018) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2018.2809722 · Cited by 477
Zhao, N., Liang, Y.-C., Niyato, D., Pei, Y., Wu, M., & Jiang, Y. (2019) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2933417 · Cited by 461 · cited 3×
“• -Greedy strategy [34]: We apply the strategy that selects a random action at with probability , and selects the action at = max a Q (st, a; θ) with probability 1 − ,”
BackgroundMethodAMP-Inspired Deep Networks for Sparse Linear Inverse Problems
Borgerding, M., Schniter, P., & Rangan, S. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2708040 · Cited by 455
Model-Driven Deep Learning for MIMO Detection
He, H., Wen, C.-K., Jin, S., & Li, G. Y. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2976585 · Cited by 454 · cited 1×
“To address these issues, so-called deep-unfolding NNs have been developed in [22]–[26], where the iterative optimization algorithms have been unfolded into a layer-wise structure that is similar to the NN.”
Millimeter Wave MIMO With Lens Antenna Array: A New Path Division Multiplexing Paradigm
Zeng, Y., & Zhang, R. (2016) · IEEE Transactions on Communications · doi:10.1109/tcomm.2016.2533490 · Cited by 356
Beamspace MIMO for high-dimensional multiuser communication at millimeter-wave frequencies
Sayeed, A. M., & Brady, J. H. (2013) · Globecom · doi:10.1109/glocom.2013.6831645 · Cited by 328
Low RF-Complexity Millimeter-Wave Beamspace-MIMO Systems by Beam Selection
Amadori, P. V., & Masouros, C. (2015) · IEEE Transactions on Communications · doi:10.1109/tcomm.2015.2431266 · Cited by 312 · cited 2×
“Then, we apply the well-known channel model [9], [12]”
MethodWei, Y., Yu, F. R., Song, M., & Han, Z. (2018) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2017.2769644 · Cited by 301 · cited 1×
“Recently, the machine learning techniques emerge as an efficient method and there are a number of studies employing the neural networks (NNs) for solving the investigated problems [16], such as precoding design [17], beam selection [18]–[20], and resource allocation [21].”
BackgroundNear-Optimal Beam Selection for Beamspace MmWave Massive MIMO Systems
Gao, X., Dai, L., Chen, Z., Wang, Z., & Zhang, Z. (2016) · IEEE Communications Letters · doi:10.1109/lcomm.2016.2544937 · Cited by 295 · cited 2×
“Then, we apply the well-known channel model [9], [12]”
MethodDeep Unfolding for Communications Systems: A Survey and Some New Directions
Balatsoukas-Stimming, A., & Studer, C. (2019) · 2019 IEEE International Workshop on Signal Processing Systems (SiPS) · doi:10.1109/sips47522.2019.9020494 · Cited by 266 · cited 1×
“To address these issues, so-called deep-unfolding NNs have been developed in [22]–[26], where the iterative optimization algorithms have been unfolded into a layer-wise structure that is similar to the NN.”
Hu, Q., Cai, Y., Shi, Q., Xu, K., Yu, G., & Ding, Z. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3033334 · Cited by 253
Self-citation5G MIMO Data for Machine Learning: Application to Beam-Selection Using Deep Learning
Klautau, A., Batista, P., Gonzalez-Prelcic, N., Wang, Y., & Heath, R. W. (2018) · 2018 Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2018.8503086 · Cited by 235 · cited 1×
“The authors in [19] have developed a dataset for investigating beam selection techniques in the scenario of vehicle-to-infrastructure.”
BackgroundSun, Y., Peng, M., & Mao, S. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2871020 · Cited by 219
Liu, X., Liu, Y., Chen, Y., & Poor, H. V. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3018823 · Cited by 208 · cited 1×
“The DQN has been widely applied in mobile offloading [28], dynamic channel access [29], radio access [30], unmanned aerial vehicle (UAV) [31], intelligent reflecting surface (IRS) [32], resource allocation [33], [34], and mobile-edge computing (MEC) [35].”
BackgroundSelf-citationCloud-Based Malware Detection Game for Mobile Devices with Offloading
Xiao, L., Li, Y., Huang, X., & Du, X. (2017) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2017.2687918 · Cited by 200 · cited 1×
“The DQN has been widely applied in mobile offloading [28], dynamic channel access [29], radio access [30], resource allocation [31], [32], and mobile-edge computing (MEC) [33].”
BackgroundProactive Resource Management for LTE in Unlicensed Spectrum: A Deep Learning Perspective
Challita, U., Dong, L., & Saad, W. (2018) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2018.2829773 · Cited by 183 · cited 1×
“The DQN has been widely applied in mobile offloading [28], dynamic channel access [29], radio access [30], unmanned aerial vehicle (UAV) [31], intelligent reflecting surface (IRS) [32], resource allocation [33], [34], and mobile-edge computing (MEC) [35].”
BackgroundData-Driven-Based Analog Beam Selection for Hybrid Beamforming Under mm-Wave Channels
Long, Y., Chen, Z., Fang, J., & Tellambura, C. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2818649 · Cited by 123 · cited 3×
“The beam selection problem could be modeled as a multi-class problem [18], where the input of the beam selection DNN is the beamspace channel and it generates the probability of each beam being selected.”
BackgroundMethodAnton-Haro, C., & Mestre, X. (2019) · IEEE Access · doi:10.1109/access.2019.2895594 · Cited by 71 · cited 2×
“Recently, the machine learning techniques emerge as an efficient method and there is a number of studies employing the neural networks (NNs) for solving the investigated problems [16], such as precoding design [17], beam selection [18]–[20], and resource allocation [21].”
BackgroundMethodShen, W., Bu, X., Gao, X., Xing, C., & Hanzo, L. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2953595 · Cited by 62 · cited 1×
“The authors in [13] have developed an energy-max algorithm for beam selection and a successive interference cancellation based precoding method in massive MIMO systems.”
MethodLow-Complexity Beam Selection Algorithms for Millimeter Wave Beamspace MIMO Systems
Pal, R., Chaitanya, A. K., & Srinivas, K. V. (2019) · IEEE Communications Letters · doi:10.1109/lcomm.2019.2902147 · Cited by 55 · cited 6×
“It is an approximation of the SINR [15], which measures the interference among the users, User Index”
MethodInfluentialDeep Unfolding for Topic Models
Chien, J.-T., & Lee, C.-H. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2677439 · Cited by 44
FALP: Fast Beam Alignment in mmWave Systems With Low-Resolution Phase Shifters
Myers, N. J., Mezghani, A., & Heath, R. W. (2019) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2941490 · Cited by 40
Tao, J., Chen, J., Xing, J., Fu, S., & Xie, J. (2020) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2020.2991878 · Cited by 36 · cited 2×
“In Algorithm 2, we propose a training method to train the two NNs jointly, which is totally different from the existing joint design scheme [37], since the gradients cannot pass”
MethodGuo, R., Cai, Y., Zhao, M., Shi, Q., Champagne, B., & Hanzo, L. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2822052 · Cited by 35 · cited 2×
“A penalty dual decompsition (PDD) algorithm has been developed in [10] for jointly optimizing the beam selection and digital precoding matrices.”
MethodSelf-citationLiu, H., Yuan, X., & Zhang, Y. A. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.2985686 · Cited by 34
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“To reduce the model and sample complexity, as well as improve the performance and interpretability, unfolded neural networks were developed in [267], [268], [269] to parameterize the iterative policy via unfolding one iteration of the existing structured algorithm into one layer of a neural network.”
MethodA Survey of Beam Management for mmWave and THz Communications Towards 6G
Xue, Q., Ji, C., Ma, S., Guo, J., Xu, Y., Chen, Q., & Zhang, W. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3361991 · Cited by 280
“In [88], the joint design of beam selection and digital precoding for mmWave MU-MIMO systems was investigated to maximize the sum-rate.”
MethodLearning Decentralized Wireless Resource Allocations With Graph Neural Networks
Wang, Z., Eisen, M., & Ribeiro, A. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3163626 · Cited by 100
“And, if the parametrization is suitably chosen, they can scale to networks with large numbers of transceivers [17]–[27].”
Sun, R., Cheng, N., Li, C., Quan, W., Zhou, H., Wang, Y., Zhang, W., & Shen, X. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3574765 · Cited by 89
“To reduce online processing time and keep interpretability, a general framework of iterative WMMSE-unfolded customized DNN is developed in matrix form for the multi-user MIMO scenario [104], [105], [116].”
Deep Unfolding Hybrid Beamforming Designs for THz Massive MIMO Systems
Nguyen, N. T., Ma, M., Lavi, O., Shlezinger, N., Eldar, Y. C., Swindlehurst, A. L., & Juntti, M. J. (2023) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2023.3322852 · Cited by 79
“The former relies mainly on the learning capability of deep neural networks (DNNs) [24], [25], [26], convolutional neural networks (CNNs) [27], [28], [29], [30], [37], [38], [39], [40], or deep reinforcement learning [41], [42] to generate HBF beamformers.”
Background
Metadata
- Authors
- Hu, Q., Liu, Y., Cai, Y., Yu, G., & Ding, Z.
- Affiliations
- Zhejiang University (CN)
- University of California, Davis (US)
- Venue
- IEEE Journal on Selected Areas in Communications
- Published
- 2021-08
- Volume
- 39
- Issue
- 8
- Pages
- 2289-2304
- ISSN
- 0733-8716 (ISSN-L)
- 1558-0008
- OpenAlex
- W3167193327
- Cited by
- 64
- References
- 41
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 11 · 2023: 18 · 2022: 11 · 2021: 5
- FWCI
- 3.31
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal on Selected Areas in Communications (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61971376, 61831004)
- Zhejiang Provincial Natural Science Foundation for Distinguished Young Scholars (LR19F010002)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Millimeter-Wave Propagation and Modeling
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Wireless Communication Technologies
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced MIMO Systems Optimization
- Keywords
- millimeter wave, multi-user MIMO, beam selection, digital precoding, deep reinforcement learning, deep unfolding, double deep Q-network, sum rate maximization, Markov decision process, neural network design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.