Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning
Open accessThe DOI matches its Crossref record; its reference list is not available.
Abstract
This article deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered:multiagent reinforcement learning(RL) andparallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has a convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To improve the communication efficiency of the distributed policy gradient for MARL, a lazily aggregated policy gradient was developed in [38] to reduce the communication rounds by only communicating informative gradients of partial agents while reusing the outdated gradients for the remaining agents.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [38]
Citations
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 67 · Cited by 67
Research works printed but not linked: 7
- omidshafiei, 0, Deep decentralized multi-task multi-agent reinforcement learning under partial observability, Proc Int Conf Mach Learn, 2681
- lowe, 0, Multi-agent actor-critic for mixed cooperative-competitive environments, Proc Adv Neural Inf Process Syst, 6382
- silver, 0, Deterministic policy gradient algorithms, Proc Int Conf Mach Learn, i-387?i-395
- chen, 0, LAG: Lazily aggregated gradient for communication-efficient distributed learning, Proc Adv Neural Inf Process Syst
- papini, 0, Stochastic variance-reduced policy gradient, Proc Int Conf Mach Learn, 4026
- mcmahan, 0, Communication-efficient learning of deep networks from decentralized data, Proc Int Conf Artif Intell Statist, 1273
- nair, 0, Massively parallel methods for deep reinforcement learning, Proc Int Conf Mach Learn Deep Learn Workshop
Metadata
- Authors
- Chen, T., Zhang, K., Giannakis, G. B., & Basar, T.
- Affiliations
- Rensselaer Polytechnic Institute (US)
- University of Illinois Urbana-Champaign (US)
- University of Minnesota (US)
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, Urbana, IL, USA
- Venue
- IEEE Transactions on Control of Network Systems
- Published
- 2022-06
- Volume
- 9
- Issue
- 2
- Pages
- 917-929
- ISSN
- 2325-5870 (ISSN-L)
- 2372-2533
- OpenAlex
- W3163894240
- Cited by
- 67
- References
- 67
- Citations by year
- 2026: 10 · 2025: 11 · 2024: 22 · 2023: 9 · 2022: 9
- FWCI
- 4.03
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Control of Network Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1 509 040, 1 508 993, 1 711 471)
- Army Research Laboratory (W911NF-17-2-0196)
- Rensselaer-IBM AI Research Collaboration
- IBM AI Horizons Network
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Keywords
- distributed reinforcement learning, policy gradient, communication efficiency, multi-agent reinforcement learning, parallel reinforcement learning, convergence rate, centralized controller, communication cost reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning. https://app.cochat.ai/papers/10.1109/tcns.2021.3078100
- The paper
- Chen, T., Zhang, K., Giannakis, G. B., & Basar, T. (2022). Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning. IEEE Transactions on Control of Network Systems. https://doi.org/10.1109/tcns.2021.3078100
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.