CoChat
Sign Up
Cited by 12Reinforcement Learning in RoboticsArtificial Intelligence in GamesPsychiatry, Mental Health, Neuroscience

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Models as Agents: Optimizing Multi-Step Predictions of Interactive Local Models in Model-Based Multi-Agent Reinforcement Learning

Open access
Wu, Zifan, Yu, Chao, Chen, Chen, Hao, Jianye, Zhuo, Hankz Hankui·2023·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v37i9.26241
Open this paper in CoChat — free accountOpen PDF

Abstract

Research in model-based reinforcement learning has made significant progress in recent years. Compared to single-agent settings, the exponential dimension growth of the joint state-action space in multi-agent systems dramatically increases the complexity of the environment dynamics, which makes it infeasible to learn an accurate global model and thus necessitates the use of agent-wise local models. However, during multi-step model rollouts, the prediction of one local model can affect the predictions of other local models in the next step. As a result, local prediction errors can be propagated to other localities and eventually give rise to considerably large global errors. Furthermore, since the models are generally used to predict for multiple steps, simply minimizing one-step prediction errors regardless of their long-term effect on other models may further aggravate the propagation of local errors. To this end, we propose Models as AGents (MAG), a multi-agent model optimization framework that reversely treats the local models as multi-step decision making agents and the current policies as the dynamics during the model rollout process. In this way, the local models are able to consider the multi-step mutual affect between each other before making predictions. Theoretically, we show that the objective of MAG is approximately equivalent to maximizing a lower bound of the true environment return. Experiments on the challenging StarCraft II benchmark demonstrate the effectiveness of MAG.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v37i9.26241

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent literature identifies notable limitations that constrain AI Agents' scalability in complex, dynamic, multi-step, and/or cooperative scenarios.

    “While AI Agents represent a significant leap in artificial intelligence capabilities, particularly in automating narrow tasks through tool-augmented reasoning, recent literature identifies notable limitations that constrain their scalability in complex, dynamic, multi-step, and/or cooperative scenarios [141-143].”

    From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [141]

    The Abstract and Introduction (page numbers unavailable) identify limitations in model-based multi-agent reinforcement learning: joint state-action spaces become difficult to model, and local prediction errors accumulate across multi-step rollouts. This supports the claim for that specific setting, but not its broader application to AI agents generally or to tool-augmented task automation. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • AI Agents have notable limitations that constrain their scalability in complex, dynamic, multi-step, and/or cooperative scenarios.

    “While AI Agents represent a significant leap in artificial intelligence capabilities, particularly in automating narrow tasks through tool-augmented reasoning, recent literature identifies notable limitations that constrain their scalability in complex, dynamic, multi-step, and/or cooperative scenarios [143]-[145].”

    From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [143]

    The Abstract and Introduction describe scalability and multi-step prediction problems in model-based multi-agent reinforcement learning: joint state-action spaces grow exponentially, and errors in interacting local models can accumulate. They do not establish the same limitations for AI agents generally or for tool-augmented reasoning agents. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Amos, B.; Stanton, S.; Yarats, D.; and Wilson, A. G. 2021. On the model-based stochastic value gradient for continu- ous reinforcement learning. In Learning for Dynamics and Control, 6-20. PMLR.
  2. Boutilier, C. 1996. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th conference on Theoretical aspects of rationality and knowl- edge, 195-210.
  3. Camacho, E. F.; and Alba, C. B. 2013. Model predictive control. Springer science & business media.
  4. Chen, X.-H.; Yu, Y.; Zhu, Z.-M.; Yu, Z.; Chen, Z.; Wang, C.; Wu, Y.; Wu, H.; Qin, R.-J.; Ding, R.; et al. 2022. Adver- sarial Counterfactual Environment Model Learning. arXiv preprint arXiv:2206.04890. · arXiv
  5. Chua, K.; Calandra, R.; McAllister, R.; and Levine, S. 2018. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural infor- mation processing systems, 31.
  6. Curi, S.; Berkenkamp, F.; and Krause, A. 2020. Efficient model-based reinforcement learning through optimistic pol- icy search and planning. Advances in Neural Information Processing Systems, 33: 14156-14170.
  7. Deisenroth, M. P.; Neumann, G.; Peters, J.; et al. 2013. A survey on policy search for robotics. Foundations and trends in Robotics, 2(1-2): 388-403.
  8. D'Oro, P.; and Jaśkowski, W. 2020. How to learn a use- ful critic? Model-based action-gradient-estimator policy op- timization. Advances in Neural Information Processing Sys- tems, 33: 313-324.
  9. Egorov, V.; and Shpilman, A. 2022. Scalable Multi-Agent Model-Based Reinforcement Learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, 381-390.
  10. Feinberg, V.; Wan, A.; Stoica, I.; Jordan, M. I.; Gonzalez, J. E.; and Levine, S. 2018. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101. · arXiv
  11. Foerster, J.; Farquhar, G.; Afouras, T.; Nardelli, N.; and Whiteson, S. 2018. Counterfactual multi-agent policy gra- dients. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  12. Fu, W.; Yu, C.; Xu, Z.; Yang, J.; and Wu, Y. 2022. Revisiting Some Common Practices in Cooperative Multi-Agent Rein- forcement Learning. In International Conference on Ma- chine Learning, 6863-6877. PMLR.
  13. Gheshlaghi Azar, M.; Munos, R.; and Kappen, H. J. 2013. Minimax PAC bounds on the sample complexity of rein- forcement learning with a generative model. Machine learn- ing, 91(3): 325-349.
  14. Hafner, D.; Lillicrap, T.; Fischer, I.; Villegas, R.; Ha, D.; Lee, H.; and Davidson, J. 2019. Learning latent dynamics for planning from pixels. In International conference on ma- chine learning, 2555-2565. PMLR.
  15. Hafner, D.; Lillicrap, T. P.; Norouzi, M.; and Ba, J. 2020. Mastering Atari with Discrete World Models. In Interna- tional Conference on Learning Representations.
  16. Janner, M.; Fu, J.; Zhang, M.; and Levine, S. 2019. When to trust your model: Model-based policy optimization. Ad- vances in Neural Information Processing Systems, 32.
  17. Kalweit, G.; and Boedecker, J. 2017. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning, 195-206.
  18. PMLR.
  19. Kim, W.; Park, J.; and Sung, Y. 2020. Communication in multi-agent reinforcement learning: Intention sharing. In In- ternational Conference on Learning Representations.
  20. Kingma, D. P.; and Welling, M. 2013. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114. · arXiv
  21. Kurutach, T.; Clavera, I.; Duan, Y.; Tamar, A.; and Abbeel, P. 2018. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592. · arXiv
  22. Liu, Y.; Wang, W.; Hu, Y.; Hao, J.; Chen, X.; and Gao, Y. 2020. Multi-agent game abstraction via graph attention neu- ral network. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 34, 7211-7218.
  23. Lowe, R.; Wu, Y. I.; Tamar, A.; Harb, J.; Pieter Abbeel, O.; and Mordatch, I. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30.
  24. Luo, F.-M.; Xu, T.; Lai, H.; Chen, X.-H.; Zhang, W.; and Yu, Y. 2022. A Survey on Model-based Reinforcement Learn- ing. arXiv preprint arXiv:2206.09328. · arXiv
  25. Luo, Y.; Xu, H.; Li, Y.; Tian, Y.; Darrell, T.; and Ma, T. 2019. Algorithmic Framework for Model-based Deep Re- inforcement Learning with Theoretical Guarantees. In ICLR (Poster).
  26. Moerland, T. M.; Broekens, J.; and Jonker, C. M. 2020. Model-based reinforcement learning: A survey. arXiv preprint arXiv:2006.16712. · arXiv
  27. Nagabandi, A.; Konolige, K.; Levine, S.; and Kumar, V. 2020. Deep dynamics models for learning dexterous ma- nipulation. In Conference on Robot Learning, 1101-1112.
  28. PMLR.
  29. Pan, F.; He, J.; Tu, D.; and He, Q. 2020. Trust the model when it is confident: Masked model-based actor- critic. Advances in neural information processing systems, 33: 10537-10546.
  30. Rao, A. V. 2009. A survey of numerical methods for opti- mal control. Advances in the Astronautical Sciences, 135(1): 497-528.
  31. Rashid, T.; Samvelyan, M.; Schroeder, C.; Farquhar, G.; Fo- erster, J.; and Whiteson, S. 2018. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning, 4295-4304.
  32. PMLR.
  33. Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; and Moritz, P. 2015. Trust region policy optimization. In International conference on machine learning, 1889-1897.
  34. PMLR.
  35. Shang, W.; Yu, Y.; Li, Q.; Qin, Z.; Meng, Y.; and Ye, J. 2019. Environment reconstruction with hidden confounders for re- inforcement learning based recommendation. In Proceed- ings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 566-576.
  36. Shapley, L. S. 1953. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100.
  37. Song, Y.; and Sun, W. 2021. Pc-mlp: Model-based reinforce- ment learning with policy cover guided exploration. In In- ternational Conference on Machine Learning, 9801-9811.
  38. PMLR.
  39. Sukhbaatar, S.; Fergus, R.; et al. 2016. Learning multiagent communication with backpropagation. Advances in neural information processing systems, 29.
  40. Sutton, R. S. 1990. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, 216- 224. Elsevier.
  41. Todorov, E.; Erez, T.; and Tassa, Y. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ interna- tional conference on intelligent robots and systems, 5026- 5033. IEEE.
  42. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At- tention is all you need. Advances in neural information pro- cessing systems, 30.
  43. Wang, J.; Ren, Z.; Liu, T.; Yu, Y.; and Zhang, C. 2021. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In In- ternational Conference on Learning Representations.
  44. Wang, W.; Yang, T.; Liu, Y.; Hao, J.; Hao, X.; Hu, Y.; Chen, Y.; Fan, C.; and Gao, Y. 2020. From few to more: Large- scale dynamic multiagent curriculum learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, 7293-7300.
  45. Wang, X.; Zhang, Z.; and Zhang, W. 2022. Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects. arXiv preprint arXiv:2203.10603. · arXiv
  46. Wu, Z.; Yu, C.; Ye, D.; Zhang, J.; Zhuo, H. H.; et al. 2021. Coordinated proximal policy optimization. Advances in Neural Information Processing Systems, 34: 26437-26448.
  47. Xu, T.; Li, Z.; and Yu, Y. 2020. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 33: 15737-15749.
  48. Zhang, W.; Wang, X.; Shen, J.; and Zhou, M. 2021. Model-based multi-agent policy optimization with adaptive opponent-wise rollouts. arXiv preprint arXiv:2105.03363. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Wu, Z., Yu, C., Chen, C., Hao, J., & Zhuo, H. H.
Affiliations
  • Sun Yat-sen University (CN)
  • Huawei Technologies (China) (CN)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2023-06-26
Volume
37
Issue
9
Pages
10435-10443
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v37i9.26241
OpenAlex
W4382464003
Cited by
12
References
5
Citations by year
2026: 4 · 2025: 5 · 2024: 3
FWCI
2.13
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Natural Science Foundation of China (62076259)
  • Sun Yat-sen University
  • Fundamental Research Funds for the Central Universities
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
model-based reinforcement learning, multi-agent reinforcement learning, local models, multi-step forecasting, error propagation, model optimization, multi-step decision-making, StarCraft II

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review