CoChat
Sign Up
PreprintCited by 35Reinforcement Learning in RoboticsMulti-Agent Systems and NegotiationArtificial Intelligence in Games

Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

Open access
Leibo, Joel Z., Duéñez-Guzmán, Edgar, Vezhnevets, Alexander Sasha, Agapiou, John P., Sunehag, Peter, Koster, Raphael, Matyas, Jayd, Beattie, Charles +2 more·2021·arXiv
DOI 10.48550/arxiv.2107.06857arXiv 2107.06857
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised-learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap, and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent's behavior constitutes (part of) another agent's environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.06857

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Compared to Melting Pot, Neural MMO is a larger environment with flexible task specifications, as opposed to a set of individual scenarios.

    “Compared to Melting Pot, Neural MMO is a larger environment with flexible task specifications, as opposed to a set of individual scenarios.”

    From Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning (2023) · cites this paper as Leibo et al., 2021

    Section 6 (p. 6) describes Melting Pot as a collection of scenarios, each consisting of a substrate and a background population; the Conclusion (p. 9) reports approximately 85 scenarios. Our copy does not establish that Neural MMO is larger or has more flexible task specifications. It also says Melting Pot can be extended with new scenarios, so the comparison should not imply that its scenarios are fixed. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In multi-principal settings, agents may not have been trained together and therefore need to generalise to new co-players and collaborators based on their prior experience.

    “in multi-principal settings, agents may not have been trained together and therefore need to generalise to new co-players and collaborators based on their prior experience (Agapiou et al., 2022; Leibo et al., 2021; Stone et al., 2010).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Leibo et al., 2021

  • The sheer number and variance of behaviours that can be exhibited by other agents means that multi-agent systems pose an especially challenging generalisation problem for individual learners.

    “Indeed, the sheer number and variance of behaviours that can be exhibited other agents means that multi-agent systems pose an especially challenging generalisation problem for individual learners (Agapiou et al., 2022; Leibo et al., 2021; Stone et al., 2010).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Leibo et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 78

  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man´ e, D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016. · arXiv
  2. Axelrod, R. On six advances in cooperation theory. Analyse &Kritik , 22(1):130-151, 2000.
  3. Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., and Mordatch, I. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528 , 2019. · arXiv
  4. Balduzzi, D., Tuyls, K., Perolat, J., and Graepel, T. Reevaluating evaluation. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 3268-3279, 2018.
  5. Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W., Perolat, J., Jaderberg, M., and Graepel, T. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning , pp. 434-443, 2019.
  6. Bansal, T., Pachocki, J., Sidor, S., Sutskever, I., and Mordatch, I. Emergent complexity via multi-agent competition. arXiv preprint arXiv:1710.03748 , 2017. · arXiv
  7. Barreto, A., Borsa, D., Hou, S., Comanici, G., Ayg¨ un, E., Hamel, P., Toyama, D., Mourad, S., Silver, D., Precup, D., et al. The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 32:13052-13062, 2019.
  8. Beattie, C., K¨ oppe, T., Du´ e˜ nez-Guzm´ an, E. A., and Leibo, J. Z. DeepMind Lab2D. arXiv preprint arXiv:2011.07027 , 2020. · arXiv
  9. Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., J´ ozefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., de Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019. · arXiv
  10. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI gym. arXiv preprint arXiv:1606.01540 , 2016. · arXiv
  11. Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., and Dragan, A. On the utility of learning about humans for human-ai coordination. arXiv preprint arXiv:1910.05789 , 2019. · arXiv
  12. Chollet, F. On the measure of intelligence. arXiv preprint arXiv:1911.01547 , 2019. · arXiv
  13. Claus, C. and Boutilier, C. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI , 1998(746-752):2, 1998.
  14. Clune, J. AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv:1905.10985 , 2019. · arXiv
  15. Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning , pp. 1282-1289. PMLR, 2019.
  16. Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., and Graepel, T. Open problems in cooperative AI. arXiv preprint arXiv:2012.08630 , 2020. · arXiv
  17. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255. Ieee, 2009.
  18. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning , pp. 1407-1416. PMLR, 2018.
  19. Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning with opponentlearning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pp. 122-130, 2018.
  20. Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C., Leibo, J. Z., and Blundell, C. Generalization of reinforcement learners with working and episodic memory. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.
  21. Harsanyi, J. C. Games with incomplete information played by 'bayesian' players, i-iii part i. the basic model. Management science , 14(3):159-182, 1967.
  22. Hessel, M., Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., and van Hasselt, H. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 3796-3803, 2019.
  23. Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. 'other-play' for zero-shot coordination. arXiv preprint arXiv:2003.02979 , 2020. · arXiv
  24. Hughes, E., Leibo, J. Z., Philips, M. G., Tuyls, K., Du´ e˜ nezGuzm´ an, E. A., Casta˜ neda, A. G., Dunning, I., Zhu, T., McKee, K. R., Koster, R., Roff, H., and Graepel, T. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 3330-3340. 2018.
  25. Hunter, D. R. et al. Mm algorithms for generalized bradleyterry models. The annals of statistics , 32(1):384-406, 2004.
  26. Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 , 2016. · arXiv
  27. Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castaneda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. Human-level performance in 3d multiplayer games with populationbased reinforcement learning. Science , 364(6443):859865, 2019.
  28. Janssen, M. A., Holahan, R., Lee, A., and Ostrom, E. Lab experiments for the study of social-ecological systems. Science , 328(5978):613-617, 2010.
  29. Juliani, A., Khalifa, A., Berges, V.-P., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., and Lange, D. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01378 , 2019. · arXiv
  30. K¨ oster, R., McKee, K. R., Everett, R., Weidinger, L., Isaac, W. S., Hughes, E., Du´ e˜ nez-Guzm´ an, E. A., Graepel, T., Botvinick, M., and Leibo, J. Z. Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences. arXiv preprint arXiv:2010.09054 , 2020. · arXiv
  31. Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., P´ erolat, J., Silver, D., and Graepel, T. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in neural information processing systems (NeurIPS) , pp. 4190-4203, 2017.
  32. Lanctot, M., Lockhart, E., Lespiau, J.-B., Zambaldi, V., Upadhyay, S., P´ erolat, J., Srinivasan, S., Timbers, F., Tuyls, K., Omidshafiei, S., Hennes, D., Morrill, D., Muller, P., Ewalds, T., Faulkner, R., Kramar, J., De Vylder, B., Saeta, B., Bradbury, J., Ding, D., Borgeaud, S., Lai, M., Schrittwieser, J., Anthony, T., Hughes, E., Danihelka, I., and Ryan-Davis, J. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453 , 2019. · arXiv
  33. Lazaridou, A. and Baroni, M. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419 , 2020. · arXiv
  34. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature , 521(7553):436-444, 2015.
  35. Leibo, J. Z., Hughes, E., Lanctot, M., and Graepel, T. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742 , 2019a. · arXiv
  36. Leibo, J. Z., Perolat, J., Hughes, E., Wheelwright, S., Marblestone, A. H., Du´ e˜ nez-Guzm´ an, E., Sunehag, P., Dunning, I., and Graepel, T. Malthusian reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pp. 1099-1107, 2019b.
  37. Lerer, A. and Peysakhovich, A. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv:1707.01068 , 2018. · arXiv
  38. Levine, S., Kleiman-Weiner, M., Schulz, L., Tenenbaum, J., and Cushman, F. The logic of universalization guides moral judgment. Proceedings of the National Academy of Sciences , 117(42):26158-26169, 2020.
  39. Littman, M. L. Markov games as a framework for multiagent reinforcement learning. In Machine learning proceedings 1994 , pp. 157-163. Elsevier, 1994.
  40. Liu, S., Lever, G., Merel, J., Tunyasuvunakool, S., Heess, N., and Graepel, T. Emergent coordination through competition. In International Conference on Learning Representations , 2018.
  41. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275 , 2017. · arXiv
  42. Lowe, R., Foerster, J., Boureau, Y.-L., Pineau, J., and Dauphin, Y. On the pitfalls of measuring emergent communication. arXiv preprint arXiv:1903.05168 , 2019. · arXiv
  43. Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562, 2018.
  44. McKee, K. R., Gemp, I., McWilliams, B., Du` e˜ nez-Guzm´ an, E. A., Hughes, E., and Leibo, J. Z. Social diversity and social preferences in mixed-motive reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pp. 869877, 2020.
  45. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pp. 19281937. PMLR, 2016.
  46. Mordatch, I. and Abbeel, P. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.
  47. Oliehoek, F. A. and Amato, C. A concise introduction to decentralized POMDPs . Springer, 2016.
  48. Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. · arXiv
  49. Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepesvari, C., Singh, S., et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568 , 2019. · arXiv
  50. Perez-Liebana, D., Hofmann, K., Mohanty, S. P., Kuno, N., Kramer, A., Devlin, S., Gaina, R. D., and Ionita, D. The multi-agent reinforcement learning in malm¨ o (marl¨ o) competition. arXiv preprint arXiv:1901.08129 , 2019. · arXiv
  51. Perolat, J., Leibo, J. Z., Zambaldi, V., Beattie, C., Tuyls, K., and Graepel, T. A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 3643-3652, 2017.
  52. Peysakhovich, A. and Lerer, A. Prosocial learning agents solve generalized stag hunts better than selfish ones. arXiv preprint arXiv:1709.02865 , 2017. · arXiv
  53. Racaniere, S., Lampinen, A., Santoro, A., Reichert, D., Firoiu, V., and Lillicrap, T. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations , 2019.
  54. Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pp. 4295-4304. PMLR, 2018.
  55. Russell, S., Dewey, D., and Tegmark, M. Research priorities for robust and beneficial artificial intelligence. Ai Magazine , 36(4):105-114, 2015.
  56. Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International conference on machine learning , pp. 1312-1320. PMLR, 2015.
  57. Shapley, L. S. Stochastic Games. In Proc. of the National Academy of Sciences of the United States of America , 1953.
  58. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.
  59. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. Nature , 550(7676):354-359, 2017.
  60. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.
  61. Song, H. F., Abdolmaleki, A., Springenberg, J. T., Clark, A., Soyer, H., Rae, J. W., Noury, S., Ahuja, A., Liu, S., Tirumala, D., Heess, N., Belov, D., Riedmiller, M., and Botvinick, M. M. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations , 2020a. URL https://openreview. net/forum?id=SylOlp4FvH .
  62. Song, Y., Wojcicki, A., Lukasiewicz, T., Wang, J., Aryan, A., Xu, Z., Xu, M., Ding, Z., and Wu, L. Arena: A general evaluation platform and building toolkit for multi-agent intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 7253-7260, 2020b.
  63. Stone, P., Kaminka, G. A., Kraus, S., Rosenschein, J. S., et al. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In AAAI , pp. 6, 2010.
  64. Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407 , 2017. · arXiv
  65. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pp. 20852087, 2018.
  66. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction . MIT press, 2018.
  67. Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 112(1-2): 181-211, 1999.
  68. Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., and Precup, D. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pp. 761-768, 2011.
  69. Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018. · arXiv
  70. Torrado, R. R., Bontrager, P., Togelius, J., Liu, J., and PerezLiebana, D. Deep reinforcement learning for general video game ai. In 2018 IEEE Conference on Computational Intelligence and Games (CIG) , pp. 1-8. IEEE, 2018.
  71. Vezhnevets, A., Wu, Y., Eckstein, M., Leblond, R., and Leibo, J. Z. Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In International Conference on Machine Learning , pp. 97339742. PMLR, 2020.
  72. Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning , pp. 3540-3549. PMLR, 2017.
  73. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575 (7782):350-354, 2019.
  74. Wang, R., Lehman, J., Clune, J., and Stanley, K. O. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 , 2019. · arXiv
  75. Wang, R. E., Wu, S. A., Evans, J. A., Tenenbaum, J. B., Parkes, D. C., and Kleiman-Weiner, M. Too many cooks: Coordinating multi-agent collaboration through inverse planning. arXiv preprint arXiv:2003.11778 , 2020. · arXiv
  76. Ye, D., Chen, G., Zhang, W., Chen, S., Yuan, B., Liu, B., Chen, J., Liu, Z., Qiu, F., Yu, H., et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 33, 2020.
  77. Zhang, A., Ballas, N., and Pineau, J. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 , 2018a. · arXiv
  78. Zhang, A., Wu, Y., and Pineau, J. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv:1811.06032 , 2018b. · arXiv

Cited by

Sorted by citations

Related

Authors
Leibo, J. Z., Duéñez-Guzmán, E., Vezhnevets, A. S., Agapiou, J. P., Sunehag, P., Koster, R., Matyas, J., Beattie, C., Mordatch, I., & Graepel, T.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2107.06857
arXiv
2107.06857
OpenAlex
W3172330035
Cited by
31
References
0
Citations by year
2025: 4 · 2024: 9 · 2023: 8 · 2022: 10
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
multi-agent reinforcement learning, melting pot, test scenario generation, agent-environment interaction, social dilemmas, reciprocity, resource sharing, task decomposition, benchmark suite

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review