Optimization for Reinforcement Learning: From a single agent to cooperative agents
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Fueled by recent advances in deep neural networks, reinforcement learning (RL) has been in the limelight because of many recent breakthroughs in artificial intelligence, including defeating humans in games (e.g., chess, Go, StarCraft), self-driving cars, smart-home automation, and service robots, among many others. Despite these remarkable achievements, many basic tasks can still elude a single RL agent. Examples abound, from multiplayer games, multirobots, cellular-antenna tilt control, traffic-control systems, and smart power grids to network management.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“By considering CSI variations [55], [275], [276] and stochastic task arrivals [84], [85], the dynamic communication and computation resource allocation problem can be formulated as a MDP, for which deep RL, a modelfree approach, can provide efficient and robust solutions [73].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Coordination of groups of mobile autonomous agents using nearest neighbor rules
Jadbabaie, A., Lin, J., & Morse, A. S. (2003) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2003.812781 · Cited by 8,339
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650 · cited 1×
“c) Development of deep networked MARL algorithms: Another interesting direction is the application of consensus-based distributed optimizations to recent deep RL algorithms, such as deep Qlearning [55], trust region policy optimization (TRPO) [56], proximal policy optimization (PPO) [57], deep deterministic policy gra…”
BackgroundSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 1×
“Note that our review is not exhaustive given the magazine limits; we suggest the interested reader to further read [6], [19], [20].”
BackgroundConstrained Consensus and Optimization in Multi-Agent Networks
Nedic, A., Ozdaglar, A., & Parrilo, P. A. (2010) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2010.2041686 · Cited by 2,086
Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2977374 · Cited by 1,306
EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization
Shi, W., Ling, Q., Wu, G., & Yin, W. (2015) · SIAM Journal on Optimization · doi:10.1137/14096668x · Cited by 1,261
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
The Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Distributed Optimization Over Time-Varying Directed Graphs
Nedic, A., & Olshevsky, A. (2015) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2364096 · Cited by 1,187
Residual Algorithms: Reinforcement Learning with Function Approximation
Baird, L. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50013-x · Cited by 918
Multiagent cooperation and competition with deep reinforcement learning
Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., & Vicente, R. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172395 · Cited by 877
On Distributed Convex Optimization Under Inequality and Equality Constraints
Zhu, M., & Martinez, S. (2012) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2011.2167817 · Cited by 759
Konda, V. R., & Tsitsiklis, J. N. (2003) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012901385691 · Cited by 709
Harnessing Smoothness to Accelerate Distributed Optimization
Qu, G., & Li, N. (2018) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2017.2698261 · Cited by 565
Fast gradient-descent methods for temporal-difference learning with linear function approximation
Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., & Wiewiora, E. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553501 · Cited by 496
Matignon, L., Laurent, G. J., & Le Fort-Piat, N. (2012) · The Knowledge Engineering Review · doi:10.1017/s0269888912000057 · Cited by 415
Asynchronous Broadcast-Based Convex Optimization Over a Network
Nedic, A. (2011) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2010.2079650 · Cited by 310 · cited 1×
“Other distributed optimization algorithms include the exact first-order algorithm for decentralized consensus optimization [46], push-sum algorithm [47] for directed graph models, and gossip-based algorithm [48], among others.”
MethodFully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
Zhang, K., Yang, Z., Liu, H., Zhang, T., & Başar, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08757 · Cited by 293
Distributed Optimization Under Adversarial Nodes
Sundaram, S., & Gharesifard, B. (2019) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2018.2836919 · Cited by 274
Distributed Primal–Dual Subgradient Method for Multiagent Optimization via Consensus Algorithms
Yuan, D., Xu, S., & Zhao, H. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2011.2160394 · Cited by 255
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-NetworksPreprint
Foerster, J., Assael, Y., de Freitas, N., & Whiteson, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02672 · Cited by 177
Kar, S., Moura, J. M. F., & Poor, H. V. (2013) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2013.2241057 · Cited by 175
Doan, T. T., Maguluri, S. T., & Romberg, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07393 · Cited by 124
Convergence Rate of Distributed Averaging Dynamics and Optimization in Networks
Nedić, A. (2015) · Foundations and Trends® in Systems and Control · doi:10.1561/2600000004 · Cited by 105
Distributed Policy Evaluation Under Multiple Behavior Strategies
Macua, S. V., Chen, J., Zazo, S., & Sayed, A. H. (2014) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2368731 · Cited by 104
Maei, H. R., & Sutton, R. S. (2010) · Proceedings of the 3d Conference on Artificial General Intelligence (AGI-10) · doi:10.2991/agi.2010.22 · Cited by 103
Finite-Time Error Bounds For Linear Stochastic Approximation and TD LearningPreprint
Srikant, R., & Ying, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00923 · Cited by 98
The world of independent learners is not markovian
Laurent, G. J., Matignon, L., & Le Fort-Piat, N. (2011) · International Journal of Knowledge-based and Intelligent Engineering Systems · doi:10.3233/kes-2010-0206 · Cited by 96
Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement LearningPreprint
Chen, Y., & Wang, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02516 · Cited by 68
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual SpacesPreprint
Mahadevan, S., Liu, B., Thomas, P. S., Dabney, W., Giguere, S., Jacek, N., Gemp, I., & Liu, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.6757 · Cited by 62
Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement LearningPreprint
Qu, C., Mannor, S., Xu, H., Qi, Y., Song, L., & Xiong, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09326 · Cited by 54
Distributed Reinforcement Learning via Gossip
Mathkar, A. S., & Borkar, V. S. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2585302 · Cited by 52
A multi-agent reinforcement learning based routing protocol for wireless sensor networks
Liang, X., Balasingham, I., & Byun, S. (2008) · International Symposium on Wireless Communication Systems · doi:10.1109/iswcs.2008.4726117 · Cited by 32
Stanković, M. S., & Stanković, S. S. (2016) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2016.7524910 · Cited by 32
Stochastic Primal-Dual Q-LearningPreprint
Jeong, N., Lee, D., & He, N. (2018) · arXiv (Cornell University) · doi:10.2139/ssrn.5403129 · Cited by 16
Self-citationPrimal-Dual Distributed Temporal Difference LearningPreprint
Lee, D., & Hu, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07918 · Cited by 9
Self-citationZhang, K., Yang, Z., Liu, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02783 · Cited by 4
Research works printed but not linked: 19
- bertsekas, 1996, Neuro-Dynamic Programming
- dai, 0, Boosting the actor with dual critic, Proc Intl Conf on Learning Representations
- puterman, 2014, Markov Decision Processes Discrete Stochastic Dynamic Programming
- dai, 0, SBEED: Convergent reinforcement learning with nonlinear function approximation, Proc Int Conf Machine Learning, 1133
- dai, 0, Learning from conditional distributions via dual embeddings, Proc Int Conf Artificial Intelligence Statistics, 1458
- sutton, 0, A convergent O(n) temporal-difference algorithm for off-policy learning with linear function approximation, Proc Advances Neural Information Processing System Conf, 1609
- lee, 0, Target-based gradient TD-learning, Proc Int Conf Machine Learning, 3713
- leibo, 0, Multi-agent reinforcement learning in sequential social dilemmas, Proc 16th Conf Autonomous Agents and MultiAgent Systems, 464
- dalal, 0, Finite sample analyses for TD(0) with function approximation, Proc AAAI Conf Artificial Intelligence, 6144
- bhandari, 2018, A finite time analysis of temporal difference learning with linear function approximation
- hong, 0, Prox-PDA: The proximal primal-dual algorithm for fast distributed nonconvex optimization and learning over networks, Proc 34th Int Conf Mach Learn, 1529
- schulman, 0, Trust region policy optimization, Proc 31st Int Conf Mach Learn, 1889
- claus, 0, The dynamics of reinforcement learning in cooperative multiagent systems, Proc 15th Nat Conf Artificial Intelligence, 746
- zhang, 2018, Finite-sample analyses for fully decentralized multi-agent reinforcement learning
- wai, 0, Multi-agent reinforcement learning via double averaging primal-dual optimization, Proc Advances in Neural Information Processing Systems, 9672
- foerster, 0, Stabilising experience replay for deep multi-agent reinforcement learning, Proc 34th Int Conf Mach Learn, 70, 1146
- lauer, 0, An algorithm for distributed reinforcement learning in cooperative multi-agent systems, Proc 17th Int Conf Machine Learning, 535
- lauer, 0, Reinforcement learning for stochastic cooperative multi-agent systems, Proc 3rd Int Joint Conf Autonomous Agents and Multiagent Systems, 3, 1516
- peshkin, 0, Learning to cooperate via policy search, Proc 16th Conf Uncertainty in Artificial Intelligence, 489
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“By considering CSI variations [275], [55], [276] and stochastic task arrivals [85], [84], the dynamic communication and computation resource allocation problem can be formulated as a MDP, for which deep RL, a model-free approach, can provide efficient and robust solutions [73].”
BackgroundSingle and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial
Feriani, A., & Hossain, E. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3063822 · Cited by 410
“In addition, the authors in [18] provide an overview of the evolution of cooperative MARL with an emphasis on distributed optimization.”
Background
Metadata
- Authors
- Lee, D., He, N., Kamalaruban, P., & Cevher, V.
- Affiliations
- Purdue University West Lafayette (US)
- University of Illinois Urbana-Champaign (US)
- École Polytechnique Fédérale de Lausanne (CH)
- Rice University (US)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2020-05
- Volume
- 37
- Issue
- 3
- Pages
- 123-135
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3023368031
- Cited by
- 113
- References
- 42
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 17 · 2023: 15 · 2022: 22 · 2021: 26 · 2020: 4
- FWCI
- 6.58
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- NSF (NSF-CRII-1755829)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Keywords
- reinforcement learning, multi-agent reinforcement learning, cooperative agents, optimization, deep reinforcement learning, multi-robot systems, traffic management, smart grid, network management
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.