CoChat
Sign Up
Cited by 1,873Reinforcement Learning in RoboticsArtificial Intelligence in GamesAdvanced Bandit Algorithms Research

Counterfactual Multi-Agent Policy Gradients

Open access
Foerster, Jakob, Farquhar, Gregory, Afouras, Triantafyllos, Nardelli, Nantas, Whiteson, Shimon·2018·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v32i1.11794
Open this paper in CoChat — free account

Abstract

Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v32i1.11794

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Classical credit-assignment methods assume cooperative objectives and honest participation.

    “Classical credit-assignment methods assume cooperative objectives and honest participation (Foerster et al., 2018), both violated under adversarial conditions.”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Foerster et al., 2018

    The Background (p. 2) defines the paper’s credit-assignment setting as fully cooperative, with agents sharing a reward function. The paper does not say that classical credit-assignment methods generally assume honest participation; its cooperative setting alone does not establish that broader claim. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Busoniu, L.; Babuska, R.; and De Schutter, B. 2008. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems Man and Cybernetics Part C Applications and Reviews 38(2):156.
  2. Cao, Y.; Yu, W.; Ren, W.; and Chen, G. 2013. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics 9(1):427-438.
  3. Chang, Y.-H.; Ho, T.; and Kaelbling, L. P. 2003. All learning is local: Multi-agent learning in global reward games. In NIPS , 807-814.
  4. Cho, K.; van Merri¨ enboer, B.; Bahdanau, D.; and Bengio, Y. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 . · arXiv
  5. Colby, M. K.; Curran, W.; and Tumer, K. 2015. Approximating difference evaluations with local information. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , 1659-1660. International Foundation for Autonomous Agents and Multiagent Systems.
  6. Collobert, R.; Kavukcuoglu, K.; and Farabet, C. 2011. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop .
  7. Das, A.; Kottur, S.; Moura, J. M.; Lee, S.; and Batra, D. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. arXiv preprint arXiv:1703.06585 . · arXiv
  8. Foerster, J.; Assael, Y . M.; de Freitas, N.; and Whiteson, S. 2016. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems , 2137-2145.
  9. Foerster, J.; Nardelli, N.; Farquhar, G.; Torr, P.; Kohli, P.; Whiteson, S.; et al. 2017. Stabilising experience replay for deep multi-agent reinforcement learning. In Proceedings of The 34th International Conference on Machine Learning .
  10. Hausknecht, M., and Stone, P. 2015. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv:1507.06527 . · arXiv
  11. Jorge, E.; K˚ ageb¨ ack, M.; and Gustavsson, E. 2016. Learning to play guess who? and inventing a grounded language as a consequence. arXiv preprint arXiv:1611.03218 . · arXiv
  12. Konda, V. R., and Tsitsiklis, J. N. 2000. Actor-critic algorithms. In Advances in neural information processing systems , 1008-1014.
  13. Lazaridou, A.; Peysakhovich, A.; and Baroni, M. 2016. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182 . · arXiv
  14. Leibo, J. Z.; Zambaldi, V.; Lanctot, M.; Marecki, J.; and Graepel, T. 2017. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037 . · arXiv
  15. Lowe, R.; Wu, Y.; Tamar, A.; Harb, J.; Abbeel, P.; and Mordatch, I. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275 . · arXiv
  16. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Humanlevel control through deep reinforcement learning. Nature 518(7540):529-533.
  17. Omidshafiei, S.; Pazis, J.; Amato, C.; How, J. P.; and Vian, J. 2017. Deep decentralized multi-task multi-agent rl under partial observability. arXiv preprint arXiv:1703.06182 . · arXiv
  18. Peng, P.; Yuan, Q.; Wen, Y.; Yang, Y.; Tang, Z.; Long, H.; and Wang, J. 2017. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069 . · arXiv
  19. Proper, S., and Tumer, K. 2012. Modeling difference rewards for multiagent learning. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 3 , 1397-1398. International Foundation for Autonomous Agents and Multiagent Systems.
  20. Schulman, J.; Moritz, P.; Levine, S.; Jordan, M. I.; and Abbeel, P. 2015. High-dimensional continuous control using generalized advantage estimation. CoRR abs/1506.02438. · arXiv
  21. Shoham, Y., and Leyton-Brown, K. 2009. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations . New York: Cambridge University Press.
  22. Sukhbaatar, S.; Fergus, R.; et al. 2016. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems , 2244-2252.
  23. Sutton, R. S.; McAllester, D. A.; Singh, S. P.; Mansour, Y.; et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In NIPS , volume 99, 1057-1063.
  24. Sutton, R. S. 1988. Learning to predict by the methods of temporal differences. Machine learning 3(1):9-44.
  25. Synnaeve, G.; Nardelli, N.; Auvolat, A.; Chintala, S.; Lacroix, T.; Lin, Z.; Richoux, F.; and Usunier, N. 2016. Torchcraft: a library for machine learning research on realtime strategy games. arXiv preprint arXiv:1611.00625 . · arXiv
  26. Tampuu, A.; Matiisen, T.; Kodelja, D.; Kuzovkin, I.; Korjus, K.; Aru, J.; Aru, J.; and Vicente, R. 2015. Multiagent cooperation and competition with deep reinforcement learning. arXiv preprint arXiv:1511.08779 . · arXiv
  27. Tan, M. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning , 330-337.
  28. Tumer, K., and Agogino, A. 2007. Distributed agent-based air traffic flow management. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems , 255. ACM.
  29. Usunier, N.; Synnaeve, G.; Lin, Z.; and Chintala, S. 2016. Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks. arXiv preprint arXiv:1609.02993 . · arXiv
  30. Weaver, L., and Tao, N. 2001. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence , 538-545. Morgan Kaufmann Publishers Inc.
  31. Weyns, D.; Helleboogh, A.; and Holvoet, T. 2005. The packet-world: A test bed for investigating situated multiagent systems. In Software Agent-Based Applications, Platforms and Development Kits . Springer. 383-408.
  32. Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229-256.
  33. Wolpert, D. H., and Tumer, K. 2002. Optimal payoff functions for members of collectives. In Modeling complexity in economic and social systems . World Scientific. 355-369.
  34. Ye, D.; Zhang, M.; and Yang, Y. 2015. A multi-agent framework for packet routing in wireless sensor networks. sensors 15(5):10026-10047.
  35. Ying, W., and Dayong, S. 2005. Multi-agent framework for third party logistics in e-commerce. Expert Systems with Applications 29(2):431-436.

Cited by

Sorted by citations

Authors
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S.
Affiliations
University of Oxford (GB)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2018-04-29
Volume
32
Issue
1
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v32i1.11794
OpenAlex
W2617547828
Cited by
1,873
References
2
Citations by year
2026: 249 · 2025: 343 · 2024: 303 · 2023: 303 · 2022: 248 · 2021: 178 · 2020: 114 · 2019: 90 · 2018: 35 · 2017: 10
FWCI
41.97
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • Oxford University Research Archive (ORA) (University of Oxford) (submittedVersion)
Funders
  • DeepMind
  • European Commission (HORIZON2020)
  • Agence Nationale de la Recherche (ANR-19-CE23-0018)
  • Engineering and Physical Sciences Research Council
  • H2020 European Research Council (637713)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
Keywords
multi-agent reinforcement learning, coma, actor-critic, counterfactual baseline, StarCraft, micromanagement, partial observability, cooperative multi-agent systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review