Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 1,264

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Open access
Zhang, Kaiqing, Yang, Zhuoran, Başar, Tamer·2021·Studies in Systems, Decision and Control
DOI 10.1007/978-3-030-60990-0_12
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-030-60990-0_12

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Research works printed but not linked: 338

  1. Brown, N., Sandholm, T.: Libratus: the superhuman AI for no-limit Poker. In: International Joint Conference on Artificial Intelligence, pp. 5226–5228 (2017) · DOI
  2. Shalev-Shwartz, S., Shammah, S., Shashua, A.: Safe, multi-agent, reinforcement learning for autonomous driving (2016). arXiv preprint arXiv:1610.03295 · arXiv
  3. Adler, J.L., Blue, V.J.: A cooperative multi-agent transportation management and route guidance system. Transp. Res. Part C: Emerg. Technol. 10(5), 433–454 (2002) · DOI · Paper page
  4. Wang, S., Wan, J., Zhang, D., Li, D., Zhang, C.: Towards smart factory for industry 4.0: a self-organized multi-agent system with big data based feedback and coordination. Comput. Netw. 101, 158–168 (2016) · DOI · Paper page
  5. Jangmin, O., Lee, J.W., Zhang, B.T.: Stock trading system using reinforcement learning with cooperative agents. In: International Conference on Machine Learning, pp. 451–458 (2002)
  6. Lee, J.W., Park, J., Jangmin, O., Lee, J., Hong, E.: A multiagent approach to $$Q $$-learning for daily stock trading. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 37(6), 864–877 (2007) · DOI
  7. Cortes, J., Martinez, S., Karatas, T., Bullo, F.: Coverage control for mobile sensing networks. IEEE Trans. Robot. Autom. 20(2), 243–255 (2004) · DOI · Paper page
  8. Choi, J., Oh, S., Horowitz, R.: Distributed learning and cooperative control for multi-agent systems. Automatica 45(12), 2802–2814 (2009) · DOI · Paper page
  9. Castelfranchi, C.: The theory of social functions: challenges for computational social science and multi-agent learning. Cogn. Syst. Res. 2(1), 5–38 (2001) · DOI
  10. Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., Graepel, T.: Multi-agent reinforcement learning in sequential social dilemmas. In: International Conference on Autonomous Agents and Multi-Agent Systems, pp. 464–473 (2017)
  11. Hernandez-Leal, P., Kartal, B., Taylor, M.E.: A survey and critique of multiagent deep reinforcement learning (2018). arXiv preprint arXiv:1810.05587 · arXiv
  12. Zazo, S., Macua, S.V., Sánchez-Fernández, M., Zazo, J.: Dynamic potential games with constraints: fundamentals and applications in communications. IEEE Trans. Signal Process. 64(14), 3806–3821 (2016) · doi:10.1109/tsp.2016.2551693
  13. Zhang, K., Yang, Z., Liu, H., Zhang, T., Başar, T.: Fully decentralized multi-agent reinforcement learning with networked agents. In: International Conference on Machine Learning, pp. 5867–5876 (2018)
  14. Subramanian, J., Mahajan, A.: Reinforcement learning in stationary mean-field games. In: International Conference on Autonomous Agents and Multi-Agent Systems, pp. 251–259 (2019)
  15. Heinrich, J., Silver, D.: Deep reinforcement learning from self-play in imperfect-information games (2016). arXiv preprint arXiv:1603.01121 · arXiv
  16. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in Neural Information Processing Systems, pp. 6379–6390 (2017)
  17. Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients (2017). arXiv preprint arXiv:1705.08926 · DOI · Paper page
  18. Gupta, J.K., Egorov, M., Kochenderfer, M.: Cooperative multi-agent control using deep reinforcement learning. In: International Conference on Autonomous Agents and Multi-Agent Systems, pp. 66–83 (2017) · DOI · Paper page
  19. Omidshafiei, S., Pazis, J., Amato, C., How, J.P., Vian, J.: Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In: International Conference on Machine Learning, pp. 2681–2690 (2017)
  20. Kawamura, K., Mizukami, N., Tsuruoka, Y.: Neural fictitious self-play in imperfect information games with many players. In: Workshop on Computer Games, pp. 61–74 (2017) · doi:10.1007/978-3-319-75931-9_5
  21. Zhang, L., Wang, W., Li, S., Pan, G.: Monte Carlo neural fictitious self-play: Approach to approximate Nash equilibrium of imperfect-information games (2019). arXiv preprint arXiv:1903.09569 · arXiv
  22. Mazumdar, E., Ratliff, L.J.: On the convergence of gradient-based learning in continuous games (2018). arXiv preprint arXiv:1804.05464 · arXiv
  23. Jin, C., Netrapalli, P., Jordan, M.I.: Minmax optimization: stable limit points of gradient descent ascent are locally optimal (2019). arXiv preprint arXiv:1902.00618 · arXiv
  24. Zhang, K., Yang, Z., Başar, T.: Policy optimization provably converges to Nash equilibria in zero-sum linear quadratic games. In: Advances in Neural Information Processing Systems (2019)
  25. Sidford, A., Wang, M., Yang, L.F., Ye, Y.: Solving discounted stochastic two-player games with near-optimal time and sample complexity (2019). arXiv preprint arXiv:1908.11071 · arXiv
  26. Arslan, G., Yüksel, S.: Decentralized Q-learning for stochastic teams and games. IEEE Trans. Autom. Control 62(4), 1545–1558 (2017) · DOI
  27. Yongacoglu, B., Arslan, G., Yüksel, S.: Learning team-optimality for decentralized stochastic control and dynamic games (2019). arXiv preprint arXiv:1903.05812 · arXiv
  28. Zhang, K., Miehling, E., Başar, T.: Online planning for decentralized stochastic control with partial history sharing. In: IEEE American Control Conference, pp. 167–172 (2019) · doi:10.23919/acc.2019.8814803
  29. Hernandez-Leal, P., Kaisers, M., Baarslag, T., de Cote, E.M.: A survey of learning in multiagent environments: dealing with non-stationarity (2017). arXiv preprint arXiv:1707.09183 · arXiv
  30. Oroojlooy Jadid, A., Hajinezhad, D.: A review of cooperative multi-agent deep reinforcement learning (2019). arXiv preprint arXiv:1908.03963 · arXiv
  31. Zhang, K., Yang, Z., Başar, T.: Networked multi-agent reinforcement learning in continuous spaces. In: IEEE Conference on Decision and Control, pp. 2771–2776 (2018) · DOI
  32. Zhang, K., Yang, Z., Liu, H., Zhang, T., Başar, T.: Finite-sample analyses for fully decentralized multi-agent reinforcement learning (2018). arXiv preprint arXiv:1812.02783 · arXiv
  33. Monahan, G.E.: State of the art-a survey of partially observable Markov decision processes: theory, models, and algorithms. Manag. Sci. 28(1), 1–16 (1982) · DOI · Paper page
  34. Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. 1. Athena Scientific, Belmont (2005)
  35. Szepesvári, C., Littman, M.L.: A unified analysis of value-function-based reinforcement-learning algorithms. Neural Comput. 11(8), 2017–2060 (1999) · DOI · Paper page
  36. Singh, S., Jaakkola, T., Littman, M.L., Szepesvári, C.: Convergence results for single-step on-policy reinforcement-learning algorithms. Mach. Learn. 38(3), 287–308 (2000) · DOI · Paper page
  37. Chang, H.S., Fu, M.C., Hu, J., Marcus, S.I.: An adaptive sampling algorithm for solving Markov decision processes. Oper. Res. 53(1), 126–139 (2005) · DOI
  38. Kocsis, L., Szepesvári, C.: Bandit based Monte-Carlo planning. In: European Conference on Machine Learning, pp. 282–293. Springer (2006) · DOI · Paper page
  39. Coulom, R.: Efficient selectivity and backup operators in Monte-Carlo tree search. In: International Conference on Computers and Games, pp. 72–83 (2006) · DOI · Paper page
  40. Agrawal, R.: Sample mean based index policies by $$O(log n)$$ regret for the multi-armed bandit problem. Adv. Appl. Probab. 27(4), 1054–1078 (1995) · DOI · Paper page
  41. Jiang, D., Ekwedike, E., Liu, H.: Feedback-based tree search for reinforcement learning. In: International Conference on Machine Learning, pp. 2284–2293 (2018)
  42. Shah, D., Xie, Q., Xu, Z.: On reinforcement learning using Monte-Carlo tree search with supervised learning: non-asymptotic analysis (2019). arXiv preprint arXiv:1902.05213 · arXiv
  43. Tsitsiklis, J.N., Van Roy, B.: Analysis of temporal-difference learning with function approximation. In: Advances in Neural Information Processing Systems, pp. 1075–1081 (1997)
  44. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
  45. Sutton, R.S., Szepesvári, C., Maei, H.R.: A convergent $$O(n)$$ algorithm for off-policy temporal-difference learning with linear function approximation. In: Advances in Neural Information Processing Systems, vol. 21(21), pp. 1609–1616 (2008)
  46. Sutton, R.S., Maei, H.R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., Wiewiora, E.: Fast gradient-descent methods for temporal-difference learning with linear function approximation. In: International Conference on Machine Learning, pp. 993–1000 (2009) · DOI · Paper page
  47. Liu, B., Liu, J., Ghavamzadeh, M., Mahadevan, S., Petrik, M.: Finite-sample analysis of proximal gradient TD algorithms. In: Conference on Uncertainty in Artificial Intelligence, pp. 504–513 (2015)
  48. Bhatnagar, S., Precup, D., Silver, D., Sutton, R.S., Maei, H.R., Szepesvári, C.: Convergent temporal-difference learning with arbitrary smooth function approximation. In: Advances in Neural Information Processing Systems, pp. 1204–1212 (2009)
  49. Dann, C., Neumann, G., Peters, J., et al.: Policy evaluation with temporal differences: a survey and comparison. J. Mach. Learn. Res. 15, 809–883 (2014)
  50. Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems, pp. 1057–1063 (2000)
  51. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992) · DOI · Paper page
  52. Baxter, J., Bartlett, P.L.: Infinite-horizon policy-gradient estimation. J. Artif. Intell. Res. 15, 319–350 (2001) · DOI · Paper page
  53. Konda, V.R., Tsitsiklis, J.N.: Actor-critic algorithms. In: Advances in Neural Information Processing Systems, pp. 1008–1014 (2000)
  54. Bhatnagar, S., Sutton, R., Ghavamzadeh, M., Lee, M.: Natural actor-critic algorithms. Automatica 45(11), 2471–2482 (2009) · DOI · Paper page
  55. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: International Conference on Machine Learning, pp. 387–395 (2014)
  56. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889–1897 (2015)
  57. Yang, Z., Zhang, K., Hong, M., Başar, T.: A finite sample analysis of the actor-critic algorithm. In: IEEE Conference on Decision and Control, pp. 2759–2764 (2018) · doi:10.1109/cdc.2018.8619440
  58. Zhang, K., Koppel, A., Zhu, H., Başar, T.: Global convergence of policy gradient methods to (almost) locally optimal policies (2019). arXiv preprint arXiv:1906.08383 · DOI
  59. Agarwal, A., Kakade, S.M., Lee, J.D., Mahajan, G.: Optimality and approximation with policy gradient methods in Markov decision processes (2019). arXiv preprint arXiv:1908.00261 · arXiv
  60. Liu, B., Cai, Q., Yang, Z., Wang, Z.: Neural proximal/trust region policy optimization attains globally optimal policy (2019). arXiv preprint arXiv:1906.10306 · arXiv
  61. Wang, L., Cai, Q., Yang, Z., Wang, Z.: Neural policy gradient methods: global optimality and rates of convergence (2019). arXiv preprint arXiv:1909.01150 · arXiv
  62. Chen, Y., Wang, M.: Stochastic primal-dual methods and sample complexity of reinforcement learning (2016). arXiv preprint arXiv:1612.02516 · arXiv
  63. Wang, M.: Primal-dual $$\pi $$ learning: sample complexity and sublinear run time for ergodic Markov decision problems (2017). arXiv preprint arXiv:1710.06100 · arXiv
  64. Shapley, L.S.: Stochastic games. Proc. Natl. Acad. Sci. 39(10), 1095–1100 (1953) · DOI · Paper page
  65. Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 157–163 (1994) · DOI · Paper page
  66. Başar, T., Olsder, G.J.: Dynamic Noncooperative Game Theory, vol. 23. SIAM, Philadelphia (1999) · doi:10.1137/1.9781611971132
  67. Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer Science & Business Media, Berlin (2012)
  68. Boutilier, C.: Planning, learning and coordination in multi-agent decision processes. In: Conference on Theoretical Aspects of Rationality and Knowledge, pp. 195–210 (1996)
  69. Lauer, M., Riedmiller, M.: An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In: International Conference on Machine Learning (2000)
  70. Yoshikawa, T.: Decomposition of dynamic team decision problems. IEEE Trans. Autom. Control 23(4), 627–632 (1978) · doi:10.1109/tac.1978.1101791
  71. Ho, Y.C.: Team decision theory and information structures. Proc. IEEE 68(6), 644–654 (1980) · DOI · Paper page
  72. Wang, X., Sandholm, T.: Reinforcement learning to play an optimal Nash equilibrium in team Markov games. In: Advances in Neural Information Processing Systems, pp. 1603–1610 (2003)
  73. Mahajan, A.: Sequential decomposition of sequential dynamic teams: applications to real-time communication and networked control systems. Ph.D. thesis, University of Michigan (2008)
  74. González-Sánchez, D., Hernández-Lerma, O.: Discrete-Time Stochastic Control and Dynamic Potential Games: The Euler-Equation Approach. Springer Science & Business Media, Berlin (2013) · doi:10.1007/978-3-319-01059-5
  75. Valcarcel Macua, S., Zazo, J., Zazo, S.: Learning parametric closed-loop policies for Markov potential games. In: International Conference on Learning Representations (2018)
  76. Kar, S., Moura, J.M., Poor, H.V.: QD-learning: a collaborative distributed strategy for multi-agent reinforcement learning through consensus + innovations. IEEE Trans. Signal Process. 61(7), 1848–1862 (2013) · DOI · Paper page
  77. Doan, T., Maguluri, S., Romberg, J.: Finite-time analysis of distributed TD (0) with linear function approximation on multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 1626–1635 (2019)
  78. Wai, H.T., Yang, Z., Wang, Z., Hong, M.: Multi-agent reinforcement learning via double averaging primal-dual optimization. In: Advances in Neural Information Processing Systems, pp. 9649–9660 (2018)
  79. Jacobson, D.: Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games. IEEE Trans. Autom. Control 18(2), 124–131 (1973) · DOI
  80. Başar, T., Bernhard, P.: H$$_\infty $$ Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach. Birkhäuser, Boston (1995)
  81. Zhang, K., Hu, B., Başar, T.: Policy optimization for $$\cal{H}_2$$ linear control with $$\cal{H}_{\infty }$$ robustness guarantee: implicit regularization and global convergence (2019). arXiv preprint arXiv:1910.09496 · arXiv
  82. Littman, M.L.: Friend-or-foe Q-learning in general-sum games. In: International Conference on Machine Learning, pp. 322–328 (2001)
  83. Lagoudakis, M.G., Parr, R.: Learning in zero-sum team Markov games using factored value functions. In: Advances in Neural Information Processing Systems, pp. 1659–1666 (2003)
  84. Osborne, M.J., Rubinstein, A.: A Course in Game Theory. MIT Press, Cambridge (1994)
  85. Shoham, Y., Leyton-Brown, K.: Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, Cambridge (2008) · DOI · Paper page
  86. Koller, D., Megiddo, N.: The complexity of two-person zero-sum games in extensive form. Games Econ. Behav. 4(4), 528–552 (1992) · DOI · Paper page
  87. Kuhn, H.: Extensive games and the problem of information. Contrib. Theory Games 2, 193–216 (1953)
  88. Zinkevich, M., Johanson, M., Bowling, M., Piccione, C.: Regret minimization in games with incomplete information. In: Advances in Neural Information Processing Systems, pp. 1729–1736 (2008)
  89. Heinrich, J., Lanctot, M., Silver, D.: Fictitious self-play in extensive-form games. In: International Conference on Machine Learning, pp. 805–813 (2015)
  90. Srinivasan, S., Lanctot, M., Zambaldi, V., Pérolat, J., Tuyls, K., Munos, R., Bowling, M.: Actor-critic policy optimization in partially observable multiagent environments. In: Advances in Neural Information Processing Systems, pp. 3422–3435 (2018)
  91. Omidshafiei, S., Hennes, D., Morrill, D., Munos, R., Perolat, J., Lanctot, M., Gruslys, A., Lespiau, J.B., Tuyls, K.: Neural replicator dynamics (2019). arXiv preprint arXiv:1906.00190 · arXiv
  92. Rubin, J., Watson, I.: Computer Poker: a review. Artif. Intell. 175(5–6), 958–987 (2011) · DOI
  93. Lanctot, M., Lockhart, E., Lespiau, J.B., Zambaldi, V., Upadhyay, S., Pérolat, J., Srinivasan, S., Timbers, F., Tuyls, K., Omidshafiei, S., et al.: Openspiel: a framework for reinforcement learning in games (2019). arXiv preprint arXiv:1908.09453 · arXiv
  94. Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: AAAI Conference on Artificial Intelligence, vol. 1998, pp. 746–752, 2p. (1998)
  95. Bowling, M., Veloso, M.: Rational and convergent learning in stochastic games. In: International Joint Conference on Artificial Intelligence, vol. 17, pp. 1021–1026 (2001)
  96. Kapetanakis, S., Kudenko, D.: Reinforcement learning of coordination in cooperative multi-agent systems. In: AAAI Conference on Artificial Intelligence, vol. 2002, pp. 326–331 (2002)
  97. Conitzer, V., Sandholm, T.: Awesome: a general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents. Mach. Learn. 67(1–2), 23–43 (2007) · DOI · Paper page
  98. Hansen, E.A., Bernstein, D.S., Zilberstein, S.: Dynamic programming for partially observable stochastic games. In: AAAI Conference on Artificial Intelligence, pp. 709–715 (2004)
  99. Amato, C., Chowdhary, G., Geramifard, A., Üre, N.K., Kochenderfer, M.J.: Decentralized control of partially observable Markov decision processes. In: IEEE Conference on Decision and Control, pp. 2398–2405 (2013) · DOI
  100. Amato, C., Oliehoek, F.A.: Scalable planning and learning for multiagent POMDPs. In: AAAI Conference on Artificial Intelligence (2015) · DOI
  101. Zinkevich, M., Greenwald, A., Littman, M.L.: Cyclic equilibria in Markov games. In: Advances in Neural Information Processing Systems, pp. 1641–1648 (2006)
  102. Bowling, M., Veloso, M.: Multiagent learning using a variable learning rate. Artif. Intell. 136(2), 215–250 (2002) · DOI · Paper page
  103. Bowling, M.: Convergence and no-regret in multiagent learning. In: Advances in Neural Information Processing Systems, pp. 209–216 (2005)
  104. Blum, A., Mansour, Y.: Learning, regret minimization, and equilibria. In: Algorithmic Game Theory, pp. 79–102 (2007) · DOI · Paper page
  105. Hart, S., Mas-Colell, A.: A reinforcement procedure leading to correlated equilibrium. In: Economics Essays, pp. 181–200. Springer, Berlin (2001) · DOI
  106. Kasai, T., Tenmoto, H., Kamiya, A.: Learning of communication codes in multi-agent reinforcement learning problem. In: IEEE Conference on Soft Computing in Industrial Applications, pp. 1–6 (2008) · DOI
  107. Kim, D., Moon, S., Hostallero, D., Kang, W.J., Lee, T., Son, K., Yi, Y.: Learning to schedule communication in multi-agent reinforcement learning. In: International Conference on Learning Representations (2019)
  108. Chen, T., Zhang, K., Giannakis, G.B., Başar, T.: Communication-efficient distributed reinforcement learning (2018). arXiv preprint arXiv:1812.03239 · arXiv
  109. Lin, Y., Zhang, K., Yang, Z., Wang, Z., Başar, T., Sandhu, R., Liu, J.: A communication-efficient multi-agent actor-critic algorithm for distributed reinforcement learning. In: IEEE Conference on Decision and Control (2019) · DOI
  110. Ren, J., Haupt, J.: A communication efficient hierarchical distributed optimization algorithm for multi-agent reinforcement learning. In: Real-World Sequential Decision Making Workshop at International Conference on Machine Learning (2019)
  111. Kim, W., Cho, M., Sung, Y.: Message-dropout: an efficient training method for multi-agent deep reinforcement learning. In: AAAI Conference on Artificial Intelligence (2019) · DOI
  112. He, H., Boyd-Graber, J., Kwok, K., Daumé III, H.: Opponent modeling in deep reinforcement learning. In: International Conference on Machine Learning, pp. 1804–1813 (2016)
  113. Grover, A., Al-Shedivat, M., Gupta, J., Burda, Y., Edwards, H.: Learning policy representations in multiagent systems. In: International Conference on Machine Learning, pp. 1802–1811 (2018)
  114. Gao, C., Mueller, M., Hayward, R.: Adversarial policy gradient for alternating Markov games. In: Workshop at International Conference on Learning Representations (2018)
  115. Li, S., Wu, Y., Cui, X., Dong, H., Fang, F., Russell, S.: Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In: AAAI Conference on Artificial Intelligence (2019) · DOI · Paper page
  116. Zhang, X., Zhang, K., Miehling, E., Basar, T.: Non-cooperative inverse reinforcement learning. In: Advances in Neural Information Processing Systems, pp. 9482–9493 (2019)
  117. Tan, M.: Multi-agent reinforcement learning: Independent vs. cooperative agents. In: International Conference on Machine Learning, pp. 330–337 (1993) · DOI · Paper page
  118. Matignon, L., Laurent, G.J., Le Fort-Piat, N.: Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems. Knowl. Eng. Rev. 27(1), 1–31 (2012) · DOI · Paper page
  119. Foerster, J., Nardelli, N., Farquhar, G., Torr, P., Kohli, P., Whiteson, S., et al.: Stabilising experience replay for deep multi-agent reinforcement learning. In: International Conference of Machine Learning, pp. 1146–1155 (2017)
  120. Tuyls, K., Weiss, G.: Multiagent learning: basics, challenges, and prospects. AI Mag. 33(3), 41 (2012)
  121. Guestrin, C., Lagoudakis, M., Parr, R.: Coordinated reinforcement learning. In: International Conference on Machine Learning, pp. 227–234 (2002)
  122. Guestrin, C., Koller, D., Parr, R.: Multiagent planning with factored MDPs. In: Advances in Neural Information Processing Systems, pp. 1523–1530 (2002)
  123. Kok, J.R., Vlassis, N.: Sparse cooperative Q-learning. In: International Conference on Machine learning, pp. 61–69 (2004) · DOI
  124. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K., et al.: Value-decomposition networks for cooperative multi-agent learning based on team reward. In: International Conference on Autonomous Agents and Multi-Agent Systems, pp. 2085–2087 (2018)
  125. Rashid, T., Samvelyan, M., De Witt, C.S., Farquhar, G., Foerster, J., Whiteson, S.: QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 681–689 (2018)
  126. Qu, G., Li, N.: Exploiting fast decaying and locality in multi-agent MDP with tree dependence structure. In: IEEE Conference on Decision and Control (2019) · DOI
  127. Mahajan, A.: Optimal decentralized control of coupled subsystems with control sharing. IEEE Trans. Autom. Control 58(9), 2377–2382 (2013) · doi:10.1109/tac.2013.2251807
  128. Foerster, J.N., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual multi-agent policy gradients. In: AAAI Conference on Artificial Intelligence (2018) · DOI · Paper page
  129. Dibangoye, J., Buffet, O.: Learning to act in decentralized partially observable MDPs. In: International Conference on Machine Learning, pp. 1233–1242 (2018)
  130. Kraemer, L., Banerjee, B.: Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing 190, 82–94 (2016) · DOI · Paper page
  131. Macua, S.V., Chen, J., Zazo, S., Sayed, A.H.: Distributed policy evaluation under multiple behavior strategies. IEEE Trans. Autom. Control 60(5), 1260–1274 (2015) · DOI
  132. Macua, S.V., Tukiainen, A., Hernández, D.G.O., Baldazo, D., de Cote, E.M., Zazo, S.: Diff-dac: Distributed actor-critic for average multitask deep reinforcement learning (2017). arXiv preprint arXiv:1710.10363 · arXiv
  133. Lee, D., Yoon, H., Hovakimyan, N.: Primal-dual algorithm for distributed reinforcement learning: distributed GTD. In: IEEE Conference on Decision and Control, pp. 1967–1972 (2018) · DOI
  134. Doan, T.T., Maguluri, S.T., Romberg, J.: Finite-time performance of distributed temporal difference learning with linear function approximation (2019). arXiv preprint arXiv:1907.12530 · arXiv
  135. Suttle, W., Yang, Z., Zhang, K., Wang, Z., Başar, T., Liu, J.: A multi-agent off-policy actor-critic algorithm for distributed reinforcement learning (2019). arXiv preprint arXiv:1903.06372 · arXiv · doi:10.1016/j.ifacol.2020.12.2021
  136. Littman, M.L.: Value-function reinforcement learning in Markov games. Cogn. Syst. Res. 2(1), 55–66 (2001) · DOI · Paper page
  137. Son, K., Kim, D., Kang, W.J., Hostallero, D.E., Yi, Y.: QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 5887–5896 (2019)
  138. Perolat, J., Piot, B., Pietquin, O.: Actor-critic fictitious play in simultaneous move multistage games. In: International Conference on Artificial Intelligence and Statistics (2018)
  139. Monderer, D., Shapley, L.S.: Potential games. Games Econ. Behav. 14(1), 124–143 (1996) · DOI · Paper page
  140. Başar, T., Zaccour, G.: Handbook of Dynamic Game Theory. Springer, Berlin (2018) · doi:10.1007/978-3-319-44374-4
  141. Huang, M., Caines, P.E., Malhamé, R.P.: Individual and mass behaviour in large population stochastic wireless power control problems: centralized and Nash equilibrium solutions. In: IEEE Conference on Decision and Control, pp. 98–103 (2003)
  142. Huang, M., Malhamé, R.P., Caines, P.E., et al.: Large population stochastic dynamic games: closed-loop Mckean-Vlasov systems and the Nash certainty equivalence principle. Commun. Inf. Syst. 6(3), 221–252 (2006) · DOI
  143. Lasry, J.M., Lions, P.L.: Mean field games. Jpn. J. Math. 2(1), 229–260 (2007) · DOI · Paper page
  144. Bensoussan, A., Frehse, J., Yam, P., et al.: Mean Field Games and Mean Field Type Control Theory, vol. 101. Springer, Berlin (2013) · DOI · Paper page
  145. Tembine, H., Zhu, Q., Başar, T.: Risk-sensitive mean-field games. IEEE Trans. Autom. Control 59(4), 835–850 (2013) · DOI
  146. Arabneydi, J., Mahajan, A.: Team optimal control of coupled subsystems with mean-field sharing. In: IEEE Conference on Decision and Control, pp. 1669–1674 (2014) · DOI
  147. Arabneydi, J.: New concepts in team theory: Mean field teams and reinforcement learning. Ph.D. thesis, McGill University (2017)
  148. Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., Wang, J.: Mean field multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 5571–5580 (2018)
  149. Witsenhausen, H.S.: Separation of estimation and control for discrete time systems. Proc. IEEE 59(11), 1557–1566 (1971) · DOI · Paper page
  150. Yüksel, S., Başar, T.: Stochastic Networked Control Systems: Stabilization and Optimization Under Information Constraints. Springer Science & Business Media, Berlin (2013) · doi:10.1007/978-1-4614-7085-4
  151. Subramanian, J., Seraj, R., Mahajan, A.: Reinforcement learning for mean-field teams. In: Workshop on Adaptive and Learning Agents at International Conference on Autonomous Agents and Multi-Agent Systems (2018)
  152. Arabneydi, J., Mahajan, A.: Linear quadratic mean field teams: optimal and approximately optimal decentralized solutions (2016). arXiv preprint arXiv:1609.00056 · arXiv
  153. Carmona, R., Laurière, M., Tan, Z.: Linear-quadratic mean-field reinforcement learning: convergence of policy gradient methods (2019). arXiv preprint arXiv:1910.04295 · arXiv
  154. Carmona, R., Laurière, M., Tan, Z.: Model-free mean-field reinforcement learning: mean-field MDP and mean-field Q-learning (2019). arXiv preprint arXiv:1910.12802 · arXiv
  155. Dall’Anese, E., Zhu, H., Giannakis, G.B.: Distributed optimal power flow for smart microgrids. IEEE Trans. Smart Grid 4(3), 1464–1475 (2013) · DOI · Paper page
  156. Zhang, K., Shi, W., Zhu, H., Dall’Anese, E., Başar, T.: Dynamic power distribution system management with a locally connected communication network. IEEE J. Sel. Top. Signal Process. 12(4), 673–687 (2018) · DOI
  157. Zhang, K., Lu, L., Lei, C., Zhu, H., Ouyang, Y.: Dynamic operations and pricing of electric unmanned aerial vehicle systems and power networks. Transp. Res. Part C: Emerg. Technol. 92, 472–485 (2018) · doi:10.1016/j.trc.2018.05.011
  158. Corke, P., Peterson, R., Rus, D.: Networked robots: flying robot navigation using a sensor net. Robot. Res. 234–243 (2005) · doi:10.1007/11008941_25
  159. Zhang, K., Liu, Y., Liu, J., Liu, M., Başar, T.: Distributed learning of average belief over networks using sequential observations. Automatica (2019) · DOI
  160. Agarwal, A., Duchi, J.C.: Distributed delayed stochastic optimization. In: Advances in Neural Information Processing Systems, pp. 873–881 (2011)
  161. Jakovetic, D., Xavier, J., Moura, J.M.: Cooperative convex optimization in networked systems: augmented Lagrangian algorithms with directed gossip communication. IEEE Trans. Signal Process. 59(8), 3889–3902 (2011) · DOI
  162. Tu, S.Y., Sayed, A.H.: Diffusion strategies outperform consensus strategies for distributed estimation over adaptive networks. IEEE Trans. Signal Process. 60(12), 6217–6234 (2012) · DOI · Paper page
  163. Varshavskaya, P., Kaelbling, L.P., Rus, D.: Efficient distributed reinforcement learning through agreement. In: Distributed Autonomous Robotic Systems, pp. 367–378 (2009) · DOI
  164. Ciosek, K., Whiteson, S.: Expected policy gradients for reinforcement learning (2018). arXiv preprint arXiv:1801.03326 · DOI
  165. Sutton, R.S., Mahmood, A.R., White, M.: An emphatic approach to the problem of off-policy temporal-difference learning. J. Mach. Learn. Res. 17(1), 2603–2631 (2016)
  166. Yu, H.: On convergence of emphatic temporal-difference learning. In: Conference on Learning Theory, pp. 1724–1751 (2015)
  167. Zhang, Y., Zavlanos, M.M.: Distributed off-policy actor-critic reinforcement learning with policy consensus (2019). arXiv preprint arXiv:1903.09255 · arXiv · doi:10.1109/cdc40024.2019.9029969
  168. Pennesi, P., Paschalidis, I.C.: A distributed actor-critic algorithm and applications to mobile sensor network coordination problems. IEEE Trans. Autom. Control 55(2), 492–497 (2010) · DOI
  169. Lange, S., Gabel, T., Riedmiller, M.: Batch reinforcement learning. In: Reinforcement Learning, pp. 45–73. Springer, Berlin (2012) · DOI · Paper page
  170. Riedmiller, M.: Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method. In: European Conference on Machine Learning, pp. 317–328 (2005) · DOI · Paper page
  171. Antos, A., Szepesvári, C., Munos, R.: Fitted Q-iteration in continuous action-space MDPs. In: Advances in Neural Information Processing Systems, pp. 9–16 (2008)
  172. Hong, M., Chang, T.H.: Stochastic proximal gradient consensus over random networks. IEEE Trans. Signal Process. 65(11), 2933–2948 (2017) · DOI
  173. Munos, R.: Performance bounds in $$\ell _p$$-norm for approximate value iteration. SIAM J. Control Optim. 46(2), 541–561 (2007) · DOI
  174. Munos, R., Szepesvári, C.: Finite-time bounds for fitted value iteration. J. Mach. Learn. Res. 9(May), 815–857 (2008)
  175. Antos, A., Szepesvári, C., Munos, R.: Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Mach. Learn. 71(1), 89–129 (2008) · DOI · Paper page
  176. Farahmand, A.M., Szepesvári, C., Munos, R.: Error propagation for approximate policy and value iteration. In: Advances in Neural Information Processing Systems, pp. 568–576 (2010)
  177. Cassano, L., Yuan, K., Sayed, A.H.: Multi-agent fully decentralized off-policy learning with linear convergence rates (2018). arXiv preprint arXiv:1810.07792 · arXiv
  178. Qu, G., Li, N.: Harnessing smoothness to accelerate distributed optimization. IEEE Trans. Control Netw. Syst. 5(3), 1245–1260 (2017) · DOI · Paper page
  179. Schmidt, M., Le Roux, N., Bach, F.: Minimizing finite sums with the stochastic average gradient. Math. Program. 162(1–2), 83–112 (2017) · DOI · Paper page
  180. Ying, B., Yuan, K., Sayed, A.H.: Convergence of variance-reduced learning under random reshuffling. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2286–2290 (2018) · doi:10.1109/icassp.2018.8461739
  181. Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Mach. Learn. 22(1–3), 123–158 (1996) · DOI · Paper page
  182. Bhandari, J., Russo, D., Singal, R.: A finite time analysis of temporal difference learning with linear function approximation. In: Conference On Learning Theory, pp. 1691–1692 (2018)
  183. Srikant, R., Ying, L.: Finite-time error bounds for linear stochastic approximation and TD learning. In: Conference on Learning Theory, pp. 2803–2830 (2019)
  184. Stanković, M.S., Stanković, S.S.: Multi-agent temporal-difference learning with linear function approximation: weak convergence under time-varying network topologies. In: IEEE American Control Conference, pp. 167–172 (2016) · DOI
  185. Stanković, M.S., Ilić, N., Stanković, S.S.: Distributed stochastic approximation: weak convergence and network design. IEEE Trans. Autom. Control 61(12), 4069–4074 (2016) · DOI
  186. Zhang, H., Jiang, H., Luo, Y., Xiao, G.: Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method. IEEE Trans. Ind. Electron. 64(5), 4091–4100 (2016) · DOI · Paper page
  187. Zhang, Q., Zhao, D., Lewis, F.L.: Model-free reinforcement learning for fully cooperative multi-agent graphical games. In: International Joint Conference on Neural Networks, pp. 1–6 (2018) · DOI
  188. Bernstein, D.S., Amato, C., Hansen, E.A., Zilberstein, S.: Policy iteration for decentralized control of Markov decision processes. J. Artif. Intell. Res. 34, 89–132 (2009) · DOI
  189. Amato, C., Bernstein, D.S., Zilberstein, S.: Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs. Auton. Agents Multi-Agent Syst. 21(3), 293–320 (2010) · DOI
  190. Liu, M., Amato, C., Liao, X., Carin, L., How, J.P.: Stick-breaking policy learning in Dec-POMDPs. In: International Joint Conference on Artificial Intelligence (2015)
  191. Dibangoye, J.S., Amato, C., Buffet, O., Charpillet, F.: Optimally solving Dec-POMDPs as continuous-state MDPs. J. Artif. Intell. Res. 55, 443–497 (2016) · DOI
  192. Wu, F., Zilberstein, S., Chen, X.: Rollout sampling policy iteration for decentralized POMDPs. In: Conference on Uncertainty in Artificial Intelligence (2010)
  193. Wu, F., Zilberstein, S., Jennings, N.R.: Monte-Carlo expectation maximization for decentralized POMDPs. In: International Joint Conference on Artificial Intelligence (2013)
  194. Amato, C., Zilberstein, S.: Achieving goals in decentralized POMDPs. In: International Conference on Autonomous Agents and Multi-Agent Systems, pp. 593–600 (2009)
  195. Banerjee, B., Lyle, J., Kraemer, L., Yellamraju, R.: Sample bounded distributed reinforcement learning for decentralized POMDPs. In: AAAI Conference on Artificial Intelligence (2012)
  196. Nayyar, A., Mahajan, A., Teneketzis, D.: Decentralized stochastic control with partial history sharing: a common information approach. IEEE Trans. Autom. Control 58(7), 1644–1658 (2013) · DOI · Paper page
  197. Arabneydi, J., Mahajan, A.: Reinforcement learning in decentralized stochastic control systems with partial history sharing. In: IEEE American Control Conference, pp. 5449–5456 (2015) · doi:10.1109/acc.2015.7172192
  198. Papadimitriou, C.H.: On inefficient proofs of existence and complexity classes. In: Annals of Discrete Mathematics, vol. 51, pp. 245–250. Elsevier (1992) · doi:10.1016/s0167-5060(08)70637-x
  199. Daskalakis, C., Goldberg, P.W., Papadimitriou, C.H.: The complexity of computing a Nash equilibrium. SIAM J. Comput. 39(1), 195–259 (2009) · DOI · Paper page
  200. Von Neumann, J., Morgenstern, O., Kuhn, H.W.: Theory of Games and Economic Behavior (commemorative edition). Princeton University Press, Princeton (2007) · DOI · Paper page

200 of 338 shown

Cited by

See all 1,264

Metadata

Authors
Zhang, K., Yang, Z., & Başar, T.
Venue
Studies in Systems, Decision and Control
Publisher
Springer International Publishing
Published
2021
Pages
321-384
ISSN
  • 2198-4182 (ISSN-L)
  • 2198-4190
DOI
10.1007/978-3-030-60990-0_12
OpenAlex
W2991046523
Cited by
1,264
References
34
FWCI
321.31
Type
book chapter
Language
en
Open access
green
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. https://app.cochat.ai/papers/10.1007/978-3-030-60990-0_12
The paper
Zhang, K., Yang, Z., & Başar, T. (2021). Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. Studies in Systems, Decision and Control. https://doi.org/10.1007/978-3-030-60990-0_12

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.