The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Evolutionary Dynamics of Multi-Agent Learning: A Survey
Open accessAbstract
The interaction of multiple autonomous agents gives rise to highly dynamic and nondeterministic environments, contributing to the complexity in applications such as automated financial markets, smart grids, or robotics. Due to the sheer number of situations that may arise, it is not possible to foresee and program the optimal behaviour for all agents beforehand. Consequently, it becomes essential for the success of the system that the agents can learn their optimal behaviour and adapt to new situations or circumstances. The past two decades have seen the emergence of reinforcement learning, both in single and multi-agent settings, as a strong, robust and adaptive learning paradigm. Progress has been substantial, and a wide range of algorithms are now available. An important challenge in the domain of multi-agent learning is to gain qualitative insights into the resulting system dynamics. In the past decade, tools and methods from evolutionary game theory have been successfully employed to study multi-agent learning dynamics formally in strategic interactions. This article surveys the dynamical models that have been derived for various multi-agent reinforcement learning algorithms, making it possible to study and compare them qualitatively. Furthermore, new learning algorithms that have been introduced using these evolutionary game theoretic tools are reviewed. The evolutionary models can be used to study complex strategic interactions. Examples of such analysis are given for the domains of automated trading in stock markets and collision avoidance in multi-robot systems. The paper provides a roadmap on the progress that has been achieved in analysing the evolutionary dynamics of multi-agent learning by highlighting the main results and accomplishments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Axelrod, R., & Hamilton, W. D. (1981) · Science · doi:10.1126/science.7466396 · Cited by 20,065
Theory of games and economic behavior
(2019) · 100 Years of Math Milestones · doi:10.1090/mbk/121/32 · Cited by 16,632
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Stochastic Processes in Physics and Chemistry
(2007) · Elsevier eBooks · doi:10.1016/b978-0-444-52965-7.x5000-4 · Cited by 8,692
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Evolutionary Games and Population Dynamics
Hofbauer, J., & Sigmund, K. (1998) · doi:10.1017/cbo9781139173179 · Cited by 5,502
SMITH, J. M., & PRICE, G. R. (1973) · Nature · doi:10.1038/246015a0 · Cited by 5,089
Theory of Games and Economic Behavior
Hawkins, D. (1945) · Philosophy of Science · doi:10.1086/286866 · Cited by 3,887
Stochastic Processes in Physics and Chemistry
Kampen, V. (1992) · Cited by 3,301
(2011) · SpringerReference · doi:10.1007/springerreference_60360 · Cited by 2,855
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235
Motion Planning in Dynamic Environments Using Velocity Obstacles
Fiorini, P., & Shiller, Z. (1998) · The International Journal of Robotics Research · doi:10.1177/027836499801700706 · Cited by 2,008
(2003) · Cited by 1,733
Fudenberg, D., Weibull, J. W., & Weibull, J. W. (1996) · The Scandinavian Journal of Economics · doi:10.2307/3440739 · Cited by 1,542
The Stag Hunt and the Evolution of Social Structure
Skyrms, B. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165228 · Cited by 1,524
Cooperative Multi-Agent Learning: The State of the Art
Panait, L., & Luke, S. (2005) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-005-2631-2 · Cited by 1,237
The dynamics of reinforcement learning in cooperative multiagent systems
Claus, C., & Boutilier, C. E. (1998) · Cited by 1,089
Nash Q-Learning for General-Sum Stochastic Games
Hu, J., Wellman, M. P., & Boutilier, C. (2003) · Cited by 923
Reinforcement Learning and Dynamic Programming Using Function Approximators
Busoniu, L., Babuska, R., De Schutter, B., & Ernst, D. (2017) · CRC Press eBooks · doi:10.1201/9781439821091 · Cited by 919
Tree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · doi:10.5555/1046920.1088690 · Cited by 854
Multiagent learning using a variable learning rate
Bowling, M., & Veloso, M. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00121-2 · Cited by 775
Gibbons, R. S. (1992) · Cited by 743
Learning Through Reinforcement and Replicator Dynamics
Börgers, T., & Sarin, R. (1997) · Journal of Economic Theory · doi:10.1006/jeth.1997.2319 · Cited by 737
Multi-agent systems in a distributed smart grid: Design and implementation
Pipattanasomporn, M., Feroze, H., & Rahman, S. (2009) · 2009 IEEE/PES Power Systems Conference and Exposition · doi:10.1109/psce.2009.4840087 · Cited by 718
Narendra, K. S., & Thathachar, M. A. L. (1974) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1974.5408453 · Cited by 668
If multi-agent learning is the answer, what is the question?
Shoham, Y., Powers, R., & Grenager, T. (2007) · Artificial Intelligence · doi:10.1016/j.artint.2006.02.006 · Cited by 410
Varieties of learning automata: an overview
Thathachar, M. A. L., & Sastry, P. S. (2002) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2002.1049606 · Cited by 382
Dearden, R., Friedman, N., & Russell, S. (1998) · Cited by 363
A Bayesian Framework for Reinforcement Learning
Strens, M. J. A. (2000) · International Conference on Machine Learning · Cited by 358
Weibull, J. W. (1996) · Cited by 348
Rational and convergent learning in stochastic games
Bowling, M., & Veloso, M. (2001) · Cited by 285
An analysis of cooperative coevolutionary algorithms
Wiegand, R. P., & Jong, K. A. D. (2004) · Cited by 259
Nash convergence of gradient dynamics in general-sum games
Singh, S. P., Kearns, M. J., & Mansour, Y. (2000) · Cited by 249
Evolutionary Function Approximation for Reinforcement Learning
Whiteson, S., & Stone, P. (2006) · Cited by 231
Conitzer, V., & Sandholm, T. (2007) · Machine Learning · doi:10.1007/s10994-006-0143-1 · Cited by 208
Convergence and No-Regret in Multiagent Learning
Bowling, M. (2004) · doi:10.7939/r3zs2kf41 · Cited by 207
Reinforcement learning of coordination in cooperative multi-agent systems
Kapetanakis, S., & Kudenko⋆, D. (2002) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.5555/777092.777145 · Cited by 192
A Stochastic Learning Model of Economic Behavior
Cross, J. G. (1973) · The Quarterly Journal of Economics · doi:10.2307/1882186 · Cited by 187
Extending Q-Learning to General Adaptive Multi-Agent Systems
Tesauro, G. J. (2003) · Cited by 187
Adaptive Load Balancing: A Study in Multi-Agent Learning
Schaerf, A., Shoham, Y., & Tennenholtz, M. (1995) · Journal of Artificial Intelligence Research · doi:10.1613/jair.121 · Cited by 185
Evolutionary dynamics on infinite strategy spaces
Oechssler, J., & Riedel, F. (2001) · Economic Theory · doi:10.1007/pl00004092 · Cited by 182
Learning, Regret Minimization, and Equilibria
Blum, A., & Mansour, Y. (2007) · Algorithmic Game Theory · doi:10.1017/cbo9780511800481.006 · Cited by 180
Multiagent Learning: Basics, Challenges, and Prospects
Tuyls, K., & Weiss, G. (2012) · AI Magazine · doi:10.1609/aimag.v33i3.2426 · Cited by 179
Self-citationFrom External to Internal Regret
Blum, A. L., & Monsour, Y. (2018) · Figshare · doi:10.1184/r1/6605876 · Cited by 163
Coordination in multiagent reinforcement learning: a Bayesian approach
Chalkiadakis, G., & Boutilier, C. E. (2003) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/860575.860689 · Cited by 158
What evolutionary game theory tells us about multiagent learning
Tuyls, K., & Parsons, S. (2007) · Artificial Intelligence · doi:10.1016/j.artint.2007.01.004 · Cited by 148
Self-citationEvolutionary game theory and multi-agent reinforcement learning
Tuyls, K., & Nowé, A. (2005) · The Knowledge Engineering Review · doi:10.1017/s026988890500041x · Cited by 134
Self-citationAn Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games
Tuyls, K., Hoen, P. J. ’., & Vanschoenwinkel, B. (2005) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-005-3783-9 · Cited by 133
Self-citationA selection-mutation model for q-learning in multi-agent systems
Tuyls, K., Verbeeck, K., & Lenaerts, T. (2003) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/860575.860687 · Cited by 131
Self-citationMay, R. M. (1981) · Nature · doi:10.1038/292291a0 · Cited by 124
Analyzing Complex Strategic Interactions in Multi-Agent Systems
Walsh, W. E., Das, R., Tesauro, G. J., & Kephart, J. O. (2002) · Cited by 120
Classes of Multiagent Q-learning Dynamics with epsilon-greedy Exploration
Wunder, M. B., Littman, M. L., & Babes, M. (2010) · International Conference on Machine Learning · Cited by 117
Decentralized learning in finite Markov chains
Wheeler, R. M., & Narendra, K. S. (1986) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1986.1104342 · Cited by 98
Dynamics of Boltzmann Q learning in two-player two-action games
Kianercy, A., & Galstyan, A. (2012) · Physical Review E · doi:10.1103/physreve.85.041145 · Cited by 96
Dynamic analysis of multiagent Q -learning with ε-greedy exploration
Gomes, E. R., & Kowalczyk, R. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553422 · Cited by 87
An Overview of Cooperative and Competitive Multiagent Learning
Hoen, P. J. ’., Tuyls, K., Panait, L., Luke, S., & Poutré, J. A. L. (2006) · Lecture notes in computer science · doi:10.1007/11691839_1 · Cited by 80
Self-citationContinuous action set learning automata for stochastic optimization
Santharam, G., SASTRY, P. S., & Thathachar, M. A. L. (1994) · Journal of the Franklin Institute · doi:10.1016/0016-0032(94)90039-6 · Cited by 76
From External to Internal Regret
BlumAvrim, & MansourYishay (2007) · Journal of Machine Learning Research · doi:10.5555/1314498.1314543 · Cited by 75
Frequency adjusted multi-agent Q-learning
Kaisers, M., & Tuyls, K. (2010) · doi:10.5555/1838206.1838250 · Cited by 70
Self-citationTheoretical Advantages of Lenient Learners: An Evolutionary Game Theoretic Perspective
Panait, L., Tuyls, K., & Luke, S. (2008) · Research Publications (Maastricht University) · Cited by 67
Self-citationStability of the replicator equation with continuous strategy space
Cressman, R. (2005) · Mathematical Social Sciences · doi:10.1016/j.mathsocsci.2005.03.001 · Cited by 59
Predicting and preventing coordination problems in cooperative Q-learning systems
Fulda, N., & Ventura, D. A. (2007) · International Joint Conference on Artificial Intelligence · Cited by 47
A decentralized approach for convention emergence in multi-agent systems
Mihaylov, M., Tuyls, K., & Nowé, A. (2013) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-013-9240-2 · Cited by 36
Self-citationKaisers, M. (2012) · Research Publications (Maastricht University) · doi:10.26481/dis.20121217mk · Cited by 34
Self-citationAn evolutionary game-theoretic analysis of poker strategies
Ponsen, M., Tuyls, K., Kaisers, M., & Ramon, J. (2009) · Entertainment Computing · doi:10.1016/j.entcom.2009.09.002 · Cited by 34
Self-citationA selection-mutation model for q-learning in multi-agent systems
Tuyls, K., Verbeeck, K., & Lenaerts, T. (2003) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/860685.860687 · Cited by 29
Self-citationSwitching dynamics of multi-agent learning
Vrancx, P., Tuyls, K., & Westra, R. L. (2008) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/ekbr3739 · Cited by 29
Self-citationEvolutionary Dynamics of Regret Minimization
Klos, T., van Ahee, G. J., & Tuyls, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15883-4_6 · Cited by 26
Self-citationKaisers, M., & Tuyls, K. (2011) · Cited by 18
Self-citationEvolutionary Dynamics of Q-Learning over the Sequence Form
Panozzo, F., Gatti, N., & Restelli, M. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.9012 · Cited by 10
Addressing the Policy-bias of Q-learning by Repeating Updates
Abdallah, S., Kaisers, M., Ito, T., Jonker, C., Gini, M., & Shehory, O. (2013) · Cited by 10
Self-citationReplicator Dynamics in Discrete and Continuous Strategy Spaces
Westra, R. L., & Tuyls, K. (2009) · Computational analysis, synthesis, and design of dynamic models series · doi:10.1201/9781420070248.ch7 · Cited by 10
Self-citationReplicator dynamics with mutations for games with a continuous strategy spacePreprint
Ruijgrok, M., & Ruijgrok, T. W. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.nlin/0505032 · Cited by 8
Evolutionary advantage of foresight in markets
Hennes, D., Bloembergen, D., Kaisers, M., Tuyls, K., & Parsons, S. (2012) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2330163.2330294 · Cited by 8
Self-citationEfficient Evolutionary Dynamics with Extensive-Form Games
Gatti, N., Panozzo, F., & Restelli, M. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8600 · Cited by 7
Cited by
Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2977374 · Cited by 1,306
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
Cooperative Multi-agent Control Using Deep Reinforcement Learning
Gupta, J. K., Egorov, M., & Kochenderfer, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-71682-4_5 · Cited by 874
Artificial Intelligence, Algorithmic Pricing, and Collusion
Calvano, E., Calzolari, G., Denicolò, V., & Pastorello, S. (2020) · American Economic Review · doi:10.1257/aer.20190623 · Cited by 708
A survey and critique of multiagent deep reinforcement learning
Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 671
Metadata
- Authors
- Bloembergen, D., Tuyls, K., Hennes, D., & Kaisers, M.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2015-08-17
- Volume
- 53
- Pages
- 659-697
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W1192553058
- Cited by
- 295
- References
- 80
- FWCI
- 17.20
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.