The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Multiagent learning using a variable learning rate
Open accessAbstract
Abstract Learning to act in a multiagent environment is a difficult problem since the normal definition of an optimal policy no longer applies. The optimal policy at any moment depends on the policies of the other agents. This creates a situation of learning a moving target. Previous learning algorithms have one of two shortcomings depending on their approach. They either converge to a policy that may not be optimal against the specific opponents' policies, or they may not converge at all. In this article we examine this learning problem in the framework of stochastic games. We look at a number of previous learning algorithms showing how they fail at one of the above criteria. We then contribute a new reinforcement learning technique using a variable learning rate to overcome these shortcomings. Specifically, we introduce the WoLF principle, “Win or Learn Fast”, for varying the learning rate. We examine this technique theoretically, proving convergence in self-play on a restricted class of iterated matrix games. We also present empirical results on a variety of more general stochastic games, in situations of self-play and otherwise, demonstrating the wide applicability of this method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Equilibrium points in n -person games
Nash, J. F. (1950) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.36.1.48 · Cited by 7,560
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
(1994) · Cited by 5,513
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
(1999) · Cited by 5,147
Dynamic Programming and Markov Processes.
Freimer, M., & Howard, R. A. (1961) · Journal of the American Statistical Association · doi:10.2307/2282268 · Cited by 3,378
Barto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
(2011) · SpringerReference · doi:10.1007/springerreference_60360 · Cited by 2,855
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614
The Theory of Learning in Games
Drew Fudenberg, B., & Levine, D. K. (1998) · MIT Press Books · Cited by 2,455
Shapley, L. S. (1953) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.39.10.1095 · Cited by 2,382
(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents
Tan, M. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50049-6 · Cited by 1,810
Fudenberg, D., Weibull, J. W., & Weibull, J. W. (1996) · The Scandinavian Journal of Economics · doi:10.2307/3440739 · Cited by 1,542
Competitive Markov Decision Processes
Filar, J., & Vrieze, K. (1996) · Springer eBooks · doi:10.1007/978-1-4612-4054-9 · Cited by 1,166
The dynamics of reinforcement learning in cooperative multiagent systems
Claus, C., & Boutilier, C. E. (1998) · Cited by 1,089
An Iterative Method of Solving a Game
Robinson, J. (1951) · The Annals of Mathematics · doi:10.2307/1969530 · Cited by 868
Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm
Hu, J., & Wellman, M. P. (1998) · Cited by 776
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms
Singh, S., Jaakkola, T., Littman, M. L., & Szepesvári, C. (2000) · Machine Learning · doi:10.1023/a:1007678930559 · Cited by 609
(1997) · An Introduction to Natural Computation · doi:10.7551/mitpress/3917.003.0017 · Cited by 492
Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems
Jaakkola, T., Singh, S. P., & Jordan, M. I. (1994) · Cited by 361
Learning to coordinate without sharing information
Sen, S., Sekaran, M., & Hale, J. C. (1994) · Cited by 314
Rational and convergent learning in stochastic games
Bowling, M., & Veloso, M. (2001) · Cited by 285
Self-citationGradient Descent for General Reinforcement Learning
Baird, L. C., & Moore, A. (1998) · neural information processing systems · Cited by 261
Nash convergence of gradient dynamics in general-sum games
Singh, S. P., Kearns, M. J., & Mansour, Y. (2000) · Cited by 249
Dynamic Programming and Markov Processes
Beranek, W., & Howard, R. A. (1961) · Technometrics · doi:10.2307/1266484 · Cited by 201
Two-person nonzero-sum games and quadratic programming
Mangasarian, O. L., & Stone, H. (1964) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(64)90021-6 · Cited by 152
An Analysis of Stochastic Game Theory for Multiagent Reinforcement Learning
Bowling, M., & Veloso, M. (2000) · Cited by 132
Self-citationReinforcement Learning in POMDP's via Direct Gradient AscentPreprint
Baxter, J., & Bartlett, P. L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.02383 · Cited by 90
On-line Learning and the Metrical Task System Problem
Blum, A. L., & Burch, C. (2000) · Machine Learning · doi:10.1023/a:1007621832648 · Cited by 74
Convergence Problems of General-Sum Multiagent Reinforcement Learning
Bowling, M. (2000) · Cited by 60
Self-citationConvergence of Gradient Dynamics with a Variable Learning Rate
Bowling, M., & Veloso, M. (2001) · Cited by 45
Self-citationLearning in dynamic noncooperative multiagent systems.
Hu, J., & Wellman, M. P. (1999) · Deep Blue (University of Michigan) · Cited by 10
Cited by
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235
“In [13] and [56], convergence is required for stability, and rationality is added as an adaptation criterion.”
InfluentialApplications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y.-C., & Kim, D. I. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916583 · Cited by 2,060
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Zhang, K., Yang, Z., & Başar, T. (2021) · Studies in Systems, Decision and Control · doi:10.1007/978-3-030-60990-0_12 · Cited by 1,264
Cooperative Multi-Agent Learning: The State of the Art
Panait, L., & Luke, S. (2005) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-005-2631-2 · Cited by 1,237
Dorri, A., Kanhere, S. S., & Jurdak, R. (2018) · IEEE Access · doi:10.1109/access.2018.2831228 · Cited by 1,043
Metadata
- Authors
- Bowling, M., & Veloso, M.
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2002-04
- Volume
- 136
- Issue
- 2
- Pages
- 215-250
- ISSN
- 0004-3702 (ISSN-L)
- OpenAlex
- W2120327309
- Cited by
- 775
- References
- 35
- FWCI
- 22.75
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.